For the most severe threat scenarios, you want multiple layers of automated defenses, and not have to rely on humans to analyze the traces of an attack weeks after your system got compromised. Many security teams use runbooks to glue together tools, processes, events, and actions for security incident response. A runbook lays out the step-by-step instructions to follow when a security incident happens, when an emerging threat surfaces, or when your security tool reports suspicious behavior.

But runbooks that stitch together glamorous security tooling are merely decorations without automated workflows for incident detection and response.

The security community agrees on many high-level best practices in such situations, but we need a single platform solution to orchestrate application security, observability, and DevOps practices. Because every situation is a little unique, Dynatrace makes it easy to create custom runbooks using Dynatrace Automations, fine-tuned to your individual business risks.

In this blog post, we’ll demonstrate how to use Dynatrace Automations to build a runbook that combats sophisticated security incidents with honeytokens and eBPF-based detection. We show an end-to-end solution, starting with deploying policies in a Kubernetes cluster and ending in a pull request assigned to the responsible team, all without manual intervention.

To demonstrate the integration of external security tools into the Dynatrace platform, we use Tetragon for eBPF-based security monitoring. Using Kyverno, we can automatically kick attackers out of our cluster with network policies and harden our configuration with a GitOps workflow to prevent the same incident from happening again.

Better, faster application protection and security investigation

Defending against threats in cloud-native environments requires advanced capabilities that can provide insight into runtime application details and speed up evidence-driven security investigations, such as:

• Runtime Vulnerability Analytics detects and analyzes vulnerabilities in the third-party dependencies of your applications, assisting you in reducing your overall risk profile.
• Runtime Application Protection defends your applications from the inside against injection attacks, even novel yet unseen zero-day attacks.
• The Security Investigator app helps security operators and security analysts to speed-up threat hunting and incident resolution, without losing data context.

Incident response and security investigations often require numerous manual steps. There is sufficient potential to automate many elements of this process. To demonstrate, we explore the following security nightmare:

• An attacker exploited a vulnerability, breached our perimeter without setting off any alerts, and managed to get access to the file system of one of our application containers.
• The attacker is now roaming freely in our container, searching for secrets in sensitive configuration files and abusing them to escalate privileges, move laterally to neighboring systems, or plant malware in our container.
• Unfortunately, we forgot to follow Kubernetes security best practices, such as using role-based access control (RBAC), deploying network policies, and setting the security context.

We’re going to write an automated workflow to detect and respond to this incident as it unfolds.

Step 1: Automating the placement of honeytokens to create strong indicators of compromise

In this sophisticated threat scenario, we don’t know exactly what events or indicators of compromise we should be looking for. Thus, we turn to a slightly different strategy: Setting up traps for attackers. We know that an attacker eventually looks for interesting files in our container. So, let’s provide some interesting files as a lure. This honeytoken strategy levels the playing field. Usually, defenders must attempt to close all security holes while attackers only need to find one open hole. With our honeytoken strategy, attackers must watch their steps not to trip over our traps.

As a lure, we place an s3_token file in path /run/secrets/eks.amazonaws.com/, which is one location for secrets that attackers commonly visit when attacking Amazon EKS clusters. When they try to access the file, it will generate an alert.

We use Kyverno, a popular policy management solution for Kubernetes, to automatically generate and mount this honeytoken into every new container in our Kubernetes cluster. Here’s the Kyverno policy YAML file we can use:

# kyverno-policy.yaml 
# 
apiVersion: kyverno.io/v1 
kind: ClusterPolicy 
metadata: 
  name: generate-honeytoken 
spec: 
  # policy also applies to existing namespaces 
  generateExisting: true 
  rules: 
    - name: generate-honeytoken 
      match: 
        any: 
          - resources: 
              kinds: 
                - Namespace 
      exclude: 
        any: 
          - resources: 
              namespaces: 
                - default 
                - kyverno 
                - kube-system 
                - kube-public 
      context: 
        - name: random-token 
          variable: 
            jmesPath: random('[a-z0-9]{6}') 
      generate: 
        kind: Secret 
        apiVersion: v1 
        name: honeytoken 
        namespace: "{{request.object.metadata.name}}" 
        # if the policy is deleted, also delete secrets 
        synchronize: true 
        data: 
          data: 
            token: "{{ random('[a-z0-9]{16}') | base64_encode(@) }}" 
--- 
apiVersion: kyverno.io/v1 
kind: ClusterPolicy 
metadata: 
  name: mount-honeytoken 
spec: 
  rules: 
    - name: mount-honeytoken 
      match: 
        any: 
          - resources: 
              kinds: 
                - Pod 
      mutate: 
        patchStrategicMerge: 
          spec: 
            volumes: 
              # depends on the generate-honeytoken policy 
              - name: honey-volume 
                secret: 
                  secretName: honeytoken 
            containers: 
              # match any image 
              - (image): "*" 
                volumeMounts: 
                  - name: honey-volume 
                    readOnly: true 
                    subPath: token 
                    mountPath: /run/secrets/eks.amazonaws.com/s3_token

Save this to file kyverno-policy.yaml and apply it in your cluster:

kubectl apply -f kyverno-policy.yaml

After applying those two cluster policies, we already solved a big pain of deception technology: The highly labor-intensive and tedious deployment of honeytokens that is often configured and maintained manually.

Step 2: Alerting with automated context enrichment

Placing honeytokens is only half of the story. We also need to receive alerts when attackers access our honeytokens. In our scenario, we chose Tetragon to detect read attempts on our honeytoken. Tetragon is a popular eBPF-based security tool for composing sophisticated security policies. We only need two policies for our case, but software engineers Kornilios Kourtis and Anastasios Papagiannis illustrate advanced techniques for file monitoring with eBPF and Tetragon in their blog post.

In our scenario, we only need two tracing policies: One to detect access attempts to the honeytoken file and one to track TCP connections, so we also find the attacker’s IP address.

# tetragon-policy.yaml 
# 
# This file was originally authored by Tetragon developers and adapted by Dynatrace. 
# - https://github.com/cilium/tetragon/blob/main/examples/tracingpolicy/filename_monitoring.yaml 
# - https://github.com/cilium/tetragon/blob/main/examples/tracingpolicy/tcp-connect.yaml 
# 
apiVersion: cilium.io/v1alpha1 
kind: TracingPolicy 
metadata: 
  name: monitor-honeytoken 
spec: 
  kprobes: 
    - call: security_file_permission 
      syscall: false 
      return: true 
      args: 
        # (struct file *) used for getting the path 
        - index: 0 
          type: file 
        # 0x04 is MAY_READ, 0x02 is MAY_WRITE 
        - index: 1 
          type: int 
      returnArg: 
        index: 0 
        type: int 
      returnArgAction: Post 
      selectors: 
        - matchArgs: 
            - index: 0 
              operator: Prefix 
              values: 
                - /run/secrets/eks.amazonaws.com/s3_token 
--- 
apiVersion: cilium.io/v1alpha1 
kind: TracingPolicy 
metadata: 
  name: monitor-tcp-connect 
spec: 
  kprobes: 
    - call: tcp_connect 
      syscall: false 
      args: 
        - index: 0 
          type: sock

Save this to file tetragon-policy.yaml and apply it in your cluster:

kubectl apply -f tetragon-policy.yaml

Enable the automatic ingest of Tetragon alerts to Dynatrace

We already installed Dynatrace on Kubernetes so that OneAgent automatically picks up logs from our cluster. Tetragon emits alerts using a shell process, which is typically noisy and not monitored in Dynatrace by default. To enable Dynatrace to automatically capture Tetragon logs in Dynatrace, navigate to Settings > Processes and containers > Declarative process grouping and add a new monitored technology for the Tetragon export-stdout process group:

• Process group display name: export-stdout
• Process group identifier: export-stdout
• Report process group: Always
• Detection rule > Select process property: Command line
• Detection rule > Condition: $contains(export-stdout)

To test if this setup is working, we open a shell in any of our containers and try reading the honeytoken that Kyverno previously placed. The honeytoken is only available in pods created after we apply the Kyverno policy.

kubectl exec -it -n your-namespace-name your-pod-name -- /bin/bash 
$ cat /run/secrets/eks.amazonaws.com/s3_token 
dyxt57xkazpw1cj5

Then, we open the Security Investigator app or the Notebooks app to track down the expected policy violation in Dynatrace:

fetch logs, from: now() - 1h 
| filter k8s.deployment.name == "tetragon" 
| parse content, "JSON:content" 
| filter content[process_kprobe][policy_name] == "monitor-honeytoken" 
| fields timestamp, content

Great, we spotted this indicator of compromise!

Step 3: Auto-remediate with network policies and GitOps

We want to automatically react to such intrusion alerts to reduce the burden on the incident response team. With the Dynatrace Platform, you can manage such security runbooks in a central and unified way.

Let’s assume an attacker tripped over our trap by reading the honeytoken. As a next step, we leverage the deep, context-aware observability insights Dynatrace has collected, block the attacker’s IP address, and inform the security team. Here’s an automation workflow that executes the following in order:

1. Listen for recent token access alerts using Dynatrace Query Language (DQL).
2. Correlate the alerts with logs on TCP connection events to find the attacker’s IP address.
3. Identify the owners of the affected resource and its associated source code repository.
4. Create a pull request in that repository that sets a new security policy to block the attacker’s IP address.
5. Create a ticket for the security team to follow-up on that incident.

Correlating token access alerts and network connections

The Tetragon alert on honeytoken access can’t provide the IP address of the related network session. But we can look for the parent process that established the TCP connection. Attackers rarely connect “directly” to your infrastructure; no port would be open to accept such a connection anyway. Instead, connections are established the other way around: Attackers connect from inside your infrastructure to their remote server, often referred to as the command-and-control (CnC) server. This attack technique is called opening a “reverse shell”. The resulting process tree typically consists of three processes: The malware itself, the shell it spawns, and the process that accesses the honeytoken.

With just three DQL queries, we find the attacker’s IP address. Note that the following queries target the structure of this attack technique and don’t hard-code process names, which attackers can easily change.

1. The first query finds the low-level process that accessed the honeytoken, for example, /bin/cat. We return the execution ID of its parent process, that is, the shell that called /bin/cat.

   fetch logs, from: now() - 10m 
   | filter k8s.deployment.name == "tetragon" 
   | parse content, "JSON:content" 
   | filter content[process_kprobe][policy_name] == "monitor-honeytoken" 
   |  fields timestamp, {content[node_name], alias: node_name}, 
           {content[process_kprobe][parent][exec_id], alias: parent_exec_id}

2. The second query finds the shell (for example, /bin/sh) by its execution ID. The only reason for this query is to get the execution ID of the shell’s parent process, that is, the process that initially spawned the shell.

   fetch logs, from: now() - 10m 
   | filter k8s.deployment.name == "tetragon" 
   | parse content, "JSON:content" 
   | filter content[node_name] == {{ result("fetch_tetragon_token_logs").records[0].node_name }} 
   | filter content[process_exec][process][exec_id] == {{ result("fetch_tetragon_token_logs").records[0].parent_exec_id }} 
   | fields timestamp, {content[node_name], alias: node_name}, 
            {content[process_exec][parent][exec_id], alias: parent_exec_id}

3. The third query finds processes that established network connections (for example, /usr/bin/python if the malware was written in Python) and filters on the execution ID returned from the previous query. The malware that spawned the shell also established the network connection. The Tetragon event also holds the remote IP address of the attacker’s server.

   fetch logs, from: now() - 10m 
   | filter k8s.deployment.name == "tetragon" 
   | parse content, "JSON:content" 
   | filter content[node_name] == {{ result("fetch_tetragon_shell_logs").records[0].node_name }} 
   | filter content[process_kprobe][policy_name] == "monitor-tcp-connect" 
   | filter content[process_kprobe][function_name] == "tcp_connect" 
   | filter content[process_kprobe][process][exec_id] == {{ result("fetch_tetragon_shell_logs").records[0].parent_exec_id }} 
   | fields timestamp,  
            {content[process_kprobe][args][0][sock_arg][daddr], alias:daddr},  
            {content[process_kprobe][process][pod][name], alias:pod_name}, 
            {content[process_kprobe][process][pod][namespace], alias:pod_namespace}, 
            {content[process_kprobe][process][pod][pod_labels], alias:pod_labels}, 
            {dt.entity.cloud_application_namespace, alias:namespace_id}

At this point, we got the intrusion event, and we got the attacker’s IP address. A typical security solution might create a generic security problem and assign it to – well, who? – some overworked security engineer who then has to dig up the component’s owner.

Automatically assigning tickets and pull requests to the correct developer teams

There is a better way: In the Kubernetes app, we can set ownership tags on our workloads. A best practice is to label your workloads already when you deploy them. A workflow action then finds the team responsible for this component and their source code repository. For demonstration, we close the loop with two more workflow actions:

• We open a pull request in the affected repository to create a network policy that blocks the attacker from accessing the cluster again based on the remote IP address Tetragon detected.
• We create a security ticket, notably, with full context information, and assigned to the correct team.

Bonus step: Deploying the security policy into the live cluster

The previous workflow follows the GitOps pattern for security remediation. That means we auto-create a pull request with a new security policy and let the developers review it. This creates a well-defined audit trail and avoids unintended disruption of live workloads.

Under some circumstances, you might want to kick out attackers as soon as you can, without any human intervention. We can utilize the newly introduced Kubernetes actions to perform cluster operations directly. We adapt the workflow as follows:

1. Directly deploy the security policy to the cluster, in addition to creating the pull request.
2. Possibly delete the pod on which we spotted the attacker to avoid further harm.

Deleting a pod sounds drastic, but if your environment is configured in a resilient manner, for example, if your application can tolerate a brief outage of some of its services, it’s a “cloud native strategy” to kick out an attacker. The Kubernetes control plane automatically re-creates a fresh (and uninfected) pod within seconds.

Workflows for security incident response on the Dynatrace platform

We demonstrated how to create a security runbook with Dynatrace Automations, orchestrating external security alerts, and custom processes:

• Tetragon as our source for eBPF-based security events.
• Kyverno to manage policies in Kubernetes.
• Grail as our storage for logs and events, which we query using the Dynatrace Query Language (DQL).
• Ownership tags to identify responsible developer teams.
• Jira and Slack integrations for workflows.
• Available soon: GitHub actions to automatically create pull requests with new security policies in the correct repository.
• Available soon: Kubernetes actions to make urgent changes to the live cluster directly.

Your tech stack might be different, your security runbooks might be different, your risk profile might be different, but with the unique, context-specific insights and automation capabilities of the Dynatrace platform, you can implement and automate it all easily and flexibly.

The post Context-aware security incident response with Dynatrace Automations and Tetragon appeared first on Dynatrace news.

Today’s security analysts have no easy job. Not only are cyberattacks increasing, but they’re also becoming more sophisticated, with tools such as WormGPT putting generative AI technology in the hands of attackers. As a result, analysts are turning to AI and TTP-based threat-hunting techniques to uncover how attackers are trying to exploit their environments.

While AIOps with generative AI will certainly empower security teams to mitigate threats faster and with greater precision, attackers will just as certainly utilize the same technology to create novel malware, more convincing phishing campaigns, and uncover high-risk zero-day vulnerabilities quicker.

Not only that, teams struggle to correlate events and alerts from a wide range of security tools, need to put them into context, and infer their risk for the business. But the industry as a whole is still hampered by a ubiquitous tool sprawl to achieve that critical mission under a barrage of alert noise.

In this blog post, we’ll use Dynatrace Security Analytics to go threat hunting, bringing together logs, traces, metrics, and, crucially, threat alerts. We use the power of DQL on Grail to derive high-level attacker tactics, techniques, and procedures (TTPs), which are much easier to interpret and act upon.

TTP-based threat hunting: Tactics, techniques, procedures

At Dynatrace, we don’t want to bombard you with alert noise and uncorrelated warnings. Instead, we want to focus on detecting and stopping attacks before they happen: In your applications, in context, at the exact line of code that is vulnerable and in use. But even when an attack happens, Dynatrace detects and blocks them in real time while providing you with rich technical details on the concrete attack procedure. Procedures describe the specific technical details that an adversary used to carry out an attack, for example, what script they ran to exploit a weakness.

When investigating advanced cyberattacks, it’s helpful to map attack procedures to attack techniques. Techniques describe the tactical goal an adversary is pursuing by executing a specific procedure. One of the most critical attack techniques within the MITRE ATT&CK® knowledge base of adversary tactics and techniques — and one example of what Dynatrace can prevent, detect, and block in real-time — is attack technique T1190, “Exploit Public-Facing Application”.

Public-facing applications can be an initial access vector an attacker could exploit to gain entry into a system. Attack tactics describe why an attacker performs an action, for example, to get that first foothold into your network.

Thinking in terms of tactics, techniques, and procedures (TTPs) brings many benefits. For example, security analysts can more easily stitch together advanced cyberattacks on an abstract level. Likewise, operation specialists can prioritize their efforts on monitoring the highest-risk tactics, and executives can better communicate the business risk.

Threat hunting and analyzing threat alerts with Dynatrace Security Analytics and Grail

Dynatrace offers Runtime Application Protection to detect a wide range of injection attacks in your applications. However, our customers often want to augment the data Dynatrace provides with data from third-party tools. Customers also want to carry out their own analysis tailored to specific use cases and forensic needs.

Dynatrace Grail is a data lakehouse that provides context-rich analytics capabilities for observability, security, and business data. You may also ingest additional data into our unified intelligence platform: One popular choice to gather fine-grained security data is Falco. Falco is an open-source, cloud-native security tool that utilizes the Linux kernel technology eBPF, to generate fine-grained networking, security, and observability events.

In the following sections, we demo the following:

1. Introduce Unguard, our insecure cloud-native microservices demo application.
2. Install Falco in AWS EKS to gather security-relevant events from all the happenings in Unguard.
3. Ingest those Falco events into Dynatrace Grail using falcosidekick.
4. Query Falco events in Dynatrace Grail, map them to TTPs, and conduct structural multi-step attack detection.

In other words, we find attacks that are composed of multiple steps by using TTPs and Dyntrace Smartscape for DQL in a way that eliminates alert noise.

First, Dynatrace OneAgent will automatically monitor and trace our infrastructure and communicate with Dynatrace. Second, we will enrich our data in the Grail data lakehouse by also ingesting Falco events using falcosidekick.

Setting up our TTP-based threat-hunting demo environment

Before we start threat hunting, we’ll first walk through how to set up the demo environment.

Introducing Unguard, our insecure cloud-native demo app

As our playground, we introduce Unguard, a microblogging demo application that embodies the challenges of modern cloud-native environments. It consists of eight services, written in at least four different languages, with countless vulnerabilities and misconfigurations. To keep it real, we have a load generator that creates benign traffic. It also generates OpenTelemetry traces.

Unguard was first introduced at DEFCON 31 by our colleagues Simon Ammer and Christoph Wedenig.

For the demonstration in this blog post, we want to deploy Unguard in AWS EKS and hunt for attacks within that environment. You can easily play around with Unguard by installing its Helm chart:

helm install unguard \ 
  oci://ghcr.io/dynatrace-oss/unguard/chart/unguard \ 
  --wait --namespace unguard --create-namespace

(Please read the Unguard README for detailed and up-to-date instructions)

This demo assumes your Kubernetes cluster is already monitored by Dynatrace. For instructions, see Set up Dynatrace on Kubernetes.

Deploy Falco and falcosidekick in AWS EKS

You can install Falco in various ways. For this demo, we installed it with the Helm chart in our AWS EKS cluster:

helm repo add falcosecurity https://falcosecurity.github.io/charts 
helm repo update 
helm install falco falcosecurity/falco --namespace falco --create-namespace

(See the Falco README for detailed and up-to-date instructions)

Next, we set up falcosidekick, which is a daemon that forwards Falco events to many possible outputs. We’re proud to announce that, with Falco version 2.29, currently in pre-release, you can now also use Dynatrace as an output.

You can use this minimal values.yaml configuration file for the Helm chart:

# values.yaml 
 
falcosidekick: 
  enabled: true 
  image: 
    tag: 2.29.0-rc.1 
  config: 
    # as of 2023-08-02, this feature is still a pre-release so we 
    # have to manually override the environment variables for now 
    extraEnv: 
      - name: DYNATRACE_APITOKEN 
        value: dt0c01.EXAMPLE_TOKEN_REPLACE_THE_ENTIRE_STRING 
      - name: DYNATRACE_APIURL 
        value: https://ENVIRONMENTID.live.dynatrace.com/api

(Please read the Helm chart README for detailed and up-to-date instructions)

We insert the apitoken we generated within Dynatrace and grant the token the scope logs.ingest. See the topic Dynatrace API – Tokens and authentication to learn more about creating tokens. As the apiurl, use the following:

Dynatrace SaaS:

https://ENVIRONMENTID.live.dynatrace.com/api

Dynatrace Managed:

https://YOURDOMAIN/e/ENVIRONMENTID/api

See the topic Environment ID to learn more about environment IDs.

Finally, we update the Falco Helm chart with this new configuration:

helm upgrade falco falcosecurity/falco -f values.yaml

If everything worked out well (check the pod logs otherwise), we are now able to successfully query Falco events with DQL. To verify, we open a new Notebook and see how Dynatrace automatically infers the fields from our events already:

fetch logs, from:now() - 5m 
| filter (event.provider == "Falco")

Observing TTPs using Dynatrace Security Analytics

For the sake of this demonstration, our internal red team unleashed a novel attack on our Unguard application. The attack lit up our Falco deployment with more than 100,000 events in 24 hours, more than 3,000 of them critical.

As security analysts, we know we can’t find sophisticated attacks by manually scrolling through thousands of audit logs and events. We need automation, full contextual knowledge of our infrastructure, and very often, domain-specific expertise from security analysts.

To get an initial overview, we can use DQL on Grail to visualize what MITRE techniques Falco observed in our infrastructure over the past 72 hours. We can summarize events using mitre.tactic or mitre.technique. These fields exist on many Falco alerts and are automatically ingested by the Dynatrace output of falcosidekick. We can explore the distribution of techniques with this query:

fetch logs, from:now() - 72h 
| filter event.provider == "Falco" and isNotNull(mitre.technique) 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| summarize event_count = count(), by:{mitre.technique}

In this example, we also observe that we can attribute most events to the following MITRE techniques:

• T1548.001
• T1083
• T1565
• T1055.008

After manually investigating these alerts, however, we conclude they’re noisy false positives. Some of our applications were treating environment variables in an insecure way or communicating with the Kubernetes API server with improperly configured service accounts. Therefore, we filtered them out with DQL.

Observability and context: Attributing reconnaissance activity to TTPs using distributed traces

So far in our TTP-based threat hunting, we’ve utilized Dyntrace Security Analytics to visualize ingested alerts from third-party tools.

But truly magical things arise when we combine this with the rich and high-quality observability data that our customers have valued since the beginning of Dynatrace. Using observability data, we can close an important security-relevant gap. Attackers often probe systems using automated scanning tools. Their many access attempts leave behind a lot of traces. Dynatrace PurePath is one of the core platform technologies that captures and analyzes those distributed traces across an entire infrastructure.

Attack sub-technique T1595.003 “Active Scanning: Wordlist Scanning” describes how attackers use scanners to learn about the many endpoints an application might expose to the internet. Typically, they use large lists with well-known path names, where many of them could be potentially vulnerable. Such wordlists often contain common path names, such as wp-admin, .git, or .htaccess. The following query looks for five indicative files and expresses how many of them match with the new recon.confidence field we set up to track wordlist-based scanners. The more matches, the more confident we can be that these requests came from a wordlist-based scanner:

fetch spans, from:now() - 72h 
| filter in(http.target, {"/wp-admin", "/.git", "/.htaccess", "/.ssh", "/cgi"}) 
| summarize { 
    recon.confidence = countDistinct(http.target) / 5, 
    recon.first_seen = min(timestamp), 
    recon.last_seen = max(timestamp) 
  }, by:{host.name, k8s.container.name, k8s.namespace.name, k8s.pod.name} 
| fieldsAdd mitre.technique = "T1595.003", mitre.tactic = "mitre_reconnaissance" 
| filter recon.confidence > 0.5

Indeed, it did find some reconnaissance attempts! This query scanned 2.5 million spans in less than 50 ms and reduced them to three comprehensible TTP records. With a conventional database, a query that scans millions of records would take many seconds to complete and require that we structure our queries up front. But Grail completes the search across millions of records in milliseconds, and automatically parses and infers the structure for us so we can just start writing queries directly.

We now know that the Envoy proxy in our environment most likely got scanned by an attacker. Let us bring all the bits and pieces together in the next sequence.

Structural multi-step attack detection with Dynatrace Security Analytics

Attackers typically perform many small steps to achieve their mission. Security experts like to think in terms of so-called kill chains, which describe the many stages of an attack. When you work with TTPs, the attack tactics represent those stages. While the MITRE ATT&CK® knowledge base describes as many as 14 tactics, we can distill this into three broad categories:

1. Land – First, hackers investigate their target, looking for an initial way to gain access and establish a foothold in your system.
2. Expand – Then, hackers typically try to escalate their privileges and move laterally within your system, compromising neighboring hosts.
3. Execute – Finally, they find their target and execute their mission.

Next in our demonstration of TTP-based threat hunting with Dynatrace Security Analytics, we’re going to show you a simple but effective strategy that can uncover such advanced attacks: Structural multi-step attack detection. This strategy is structural since it utilizes Smartscape for DQL to take the topological relationship of events into account when hunting for attacks that are composed of multiple steps.

The following DQL query looks for the filtered Falco alerts, and for each Kubernetes pod, it records how many distinct tactics and techniques we just observed. This way, we’re not just looking at whatever pod was the noisiest, but instead, which pod generated alerts from the most tactics and techniques. The more tactics and techniques, the higher the chances that an attacker carried out a full kill chain on that pod.

fetch logs, from:now() - 72h 
| filter (event.provider == "Falco") 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| summarize { 
    num_tactics = countDistinct(mitre.tactic), 
    num_techniques = countDistinct(mitre.technique), 
    tactics = collectDistinct(mitre.tactic), 
    techniques = collectDistinct(mitre.technique) 
  }, by:{k8s.pod.name} 
| filter num_tactics > 1 and num_techniques > 1 
| sort num_tactics desc, num_techniques desc

This result is highly interesting and confirms our previous suspicion. There is one instance of the Envoy proxy that captured alerts for the following techniques:

• TA0002 “Execution”
• TA0003 “Persistence”
• TA0005 “Defensive Evasion”
• TA0006 “Credential Access”
• TA0011 “Command and Control”
• TA0010 “Exfiltration”

Further, the Dynatrace spans we looked at in the previous sequence that explored wordlist scanning indicated TA0043 “Reconnaissance”. This query just scanned through more than 23 million records in 300 ms, providing us with an abstract description of a full kill chain.

But we don’t stop here. We can drill down and observe the individual steps our attacker has taken:

fetch logs, from:now() - 72h 
| filter (event.provider == "Falco") 
| filterOut in(mitre.technique, {"T1548.001", "T1083", "T1565", "T1055.008"}) 
| filter k8s.pod.name == "unguard-envoy-proxy-666464f76d-5p26f" 
| fields timestamp, event.name, mitre.technique, content.output_fields.proc.cmdline 
| sort timestamp asc

In the result, we see the records that explain the attack procedure in detail. The above screenshot shows only an excerpt of all 47 records. Here’s what we learned about our attacker’s steps:

• Scanned our Envoy proxy with a well-known wordlist, as we learned by mining the traces for wordlist entries.
• Launched a Perl-based reverse shell on Envoy, indicated by the perl command opening a socket, giving them full code execution access.
• Downloaded a couple of binaries like nmap and nc, indicated by the curl command that pulled them from the internet.
• Scanned our internal network with nmap.
• Exfiltrated large volumes of data from our Redis database, indicated by the queries to Redis that request all keys with the KEYS * command
• Tried to cover their tracks by deleting the shell history, indicated by the rm /home/envoy/.bash_history command

Isn’t this a truly elegant way to hunt for attacks?

TTP-based threat hunting with context-rich observability and security analytics

This demonstration shows how modern attack detection strategies become a reality with context-rich security analytics on a unified observability and security platform. With this approach, you can do the following:

• Utilize observability data to capture security-relevant reconnaissance alerts and map them to TTPs.
• Enrich the Dynatrace platform with more data of your own, such as ingesting Falco alerts into Grail.
• Use DQL and Grail to find the needle in the haystack to scan tens of millions of records in milliseconds to identify the chain of only a handful of events that exposed an attacker and the exact methods they used.

For another great demonstration, we recommend reading the blog post Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail by Liisa Tallinn.

The post TTP-based threat hunting with Dynatrace Security Analytics and Falco Alerts solves alert noise appeared first on Dynatrace news.