Solving incidents or finding root causes is a time-critical activity that requires logged evidence to understand what really happened in a system. Whether to prevent such incidents from happening again or to rule out a malicious hacking attempt, getting answers is the key, and logs are your best source for getting such evidence.

However, analyzing only logs might not be enough. For example, if the incident resulted in a system crash, it would be important to also see the utilization of the system just before the incident occurred or to understand what happened in the system around utilization peaks.

This is why Dynatrace has introduced a new advanced investigation feature for Security Investigator that gives you faster insights into Performance Metrics.

Solving latency issues with metric insights

Dynatrace Security Investigator is one of the built-in apps that ships with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into the Dynatrace Grail® data lakehouse.

Imagine you’re performing root cause analysis on a long-running query using Security Investigator. You’ve found the logs for the request that was running for a long time, and you want to understand what the CPU utilization of the application was at the time of the request.

By simply right-clicking on the log record in your results, you can choose either the relevant pod, container, host, or any other dimension available in the log record and choose CPU utilization.

As a result, you can see the CPU utilization chart from the specific pod around the time of the log record in the context of your investigation next to your logs. You can modify the aggregation function or change the visible metric for the pod, if needed.

The red indicator on the chart shows the selected log record’s timestamp from your results table. If you choose any of the other log records in the results table, the red indicator will change its position on the chart, enabling you to update the CPU utilization chart with your log-analysis context.

Persisted throughout the case

When navigating to other query nodes in your query tree, you can still see the CPU utilization chart, enabling you to analyse other logs in the same context. For example, you might want to analyze your network flow logs from the same period and see how some network requests from the same time might be connected to the pod’s CPU usage. Selecting network flow events moves the red indicator’s position on the chart (shown as a dashed line due to different entity values).

Get started

Visit the Dynatrace Playground to see performance metrics in action or learn more about how performance metrics add relevant context to your security investigations in Dynatrace Documentation.

The post Enhanced incident response based on performance-metric insights appeared first on Dynatrace news.

Log data alone, however, is often not enough to give you a complete overview. In the case of security incidents, the origin of a failed request or a login attempt is often the most valuable information you have in triaging a potential incident and determining its severity.

To speed up your threat hunting and incident response activities, Security Investigator now allows you to enrich IP addresses with additional context from third-party threat detection databases.

Fast insights into IP addresses

Dynatrace Security Investigator is one of the built-in apps shipped with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into the Dynatrace Grail® data Lakehouse.

Imagine you’re solving a security incident and need to understand more about the origin of an attack. You want to check the IP address reputation analysis provided by AbuseIPDB or VirusTotal, but manually looking up and pasting these addresses takes too much time. Not to mention, you want to keep this external reputation information tied to your investigation for future access.

By right-clicking an IP address in Security Investigator and selecting Enrich IP, you can choose your configured enrichment source to enrich the address. For this first release, Dynatrace offers IP enrichment insights from AbuseIPDB and VirusTotal.

Suppose you find something suspicious in your log data related to a certain IP address. You can now add the IP address and additional reputation analysis to your Suspicious IP evidence list in Security Investigator.

The IP address and corresponding enrichment data are now kept in the context of your investigation; you can revisit this enrichment data at any time.

Automatic IP evidence enrichment

You can now automatically enrich all the IP addresses you collect with evidence. In the evidence list enrichment menu, choose if enrichment should be enabled for all added elements by default, and which connection should be used to enrich the IP addresses that are added to your evidence list.

When you add new IP addresses to the evidence list, they’re automatically enriched on your behalf using your selected enrichment connection.

Select the details icon to quickly access the persisted IP enrichment data that’s fetched from each third-party source.

The post Security Investigator offers reputation analysis and context for IP addresses appeared first on Dynatrace news.

Dynatrace introduces a new advanced way to enhance your DQL queries in Security Investigator cases. The pivoting queries concept allows engineers to quickly change the investigation context by switching the scope of a query using available pivoting dimensions.

Time is critical

Imagine you’re investigating latency issues in your cloud applications by analyzing your Istio proxy logs with Security Investigator. You have hundreds of pods, each of which has application containers and Istio containers running side-by-side. You analyze all the Istio logs at once to get a comprehensive overview of the whole infrastructure, fetching logs for all long-running requests, and you want to investigate some of the requests to identify the issues behind the high latency.

Dynatrace Security Investigator is one of the built-in apps shipped with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into the Dynatracer Grail® data lakehouse.

Security Investigator allows you to:

• Keep your whole investigation flow in context.
• Perform complex security investigations on the data stored in Grail.
• Build DQL queries based on your findings in a fast and easy way.
• Save and use the found evidence to build your DQL queries and find answers to your questions.
• Navigate with ease to any point in your investigation history and review queries and results.
• Fetch detailed results in the original format to quickly understand the information.

Quickly shift your investigative perspective

Thanks to pivoting queries, it‘s now possible to quickly shift your investigation from one perspective to another. You can choose Pivot query by right-clicking any record in the results table in Security Investigator and selecting the dimension you want to pivot your query by, for example, Kubernetes pod. As a result, a new query node is created, containing all the logs from the same Kubernetes pod from which the Istio record originated, giving you the full context and all application logs from that pod.

Pivoting queries also work at scale: if you select multiple records and use pivoting queries with multiple values, multiple nodes will be created based on the distinct selected values. So, if you choose five result records and pivot query by trace_id, then five new query nodes will be created, which will contain results for each respective trace ID. This allows you to continue investigating each trace in your query branch, keeping the context and tracking your individual query history.

The pivoting dimensions are chosen from the metadata fields of your log records and can be further customized in Security Investigator. You can switch dimensions from, for example, a high-level Kubernetes cluster or host to the corresponding process group instance or cloud application, depending on your investigation’s purpose and the context you need.

The pivoting queries feature allows you to speed up your investigations and apply them in different contexts using the power of the query tree and other investigative features of Security Investigator.

Get started

The post Pivot the perspective of your investigative queries with Security Investigator appeared first on Dynatrace news.

Dynatrace introduces a new way to think about timestamps in your Security Investigator cases. The reference time concept uses relative timestamps to speed up incident response time.

Time is critical

Imagine an engineer performing root cause analysis by analyzing your application logs using Security Investigator. They discover the event in the logs that caused an incident, and as the next step, they want to analyze network flow logs from your Cloud Service Provider (CSP). Since network flow logs don’t contain any meaningful data about the request content or its response, it’s relatively difficult to understand which events in the network flow logs occurred before the incident and which ones followed it.

Dynatrace Security Investigator is one of the built-in apps shipped with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into Grail.

Security Investigator allows you to:

• Keep your whole investigation flow in context.
• Perform complex security investigations on the data stored in Grail.
• Build DQL queries based on your findings in a fast and usable way.
• Save and use found evidence to build your DQL queries and find answers to your questions.
• Navigate with ease to any point in your investigation history and review queries and results.
• Fetch detailed results in the original format to quickly understand the information.

Thanks to reference time, it‘s now possible to add the time perspective to keep track of the relative time between events you’re analyzing and the time that incidents occurred.

You can choose Set as reference time by right-clicking any Security Investigator timestamp field related to an event that caused an incident.

A new virtual column called timestamp_diff is then added to all the query nodes of the query tree. The virtual node contains the time difference between the set reference time and the timestamp of the event in the results table. So, even if you open another query node, you’ll still see the reference time offset field in the results menu displaying the time distance between the event and the set reference time.

The virtual offset column is created automatically for the first timestamp field in the results table (usually called timestamp), but it can be turned on for any timestamp field. You can do this from the column header menu or from the reference time menu.

This small, yet powerful feature allows engineers to speed up investigations and lock down event timestamps for easier navigation across their query results.

With reference time, you can efficiently navigate logs while maintaining an incident’s time context. It helps you track the time offset between events you’re analyzing and when the incident occurred. This allows you to uncover relevant threads and evidence, even from logs and events that might initially seem unrelated.

Reference time integrates diverse information, ensuring a consistent incident context across all data points.

For a more detailed example of how reference time speeds up threat hunting, please see the reference time use case in Dynatrace Documentation.

© 2025 Dynatrace LLC

Dynatrace and the Dynatrace logo, are trademarks of the Dynatrace, Inc. group of companies. All other trademarks are the property of their respective owners.

The post Create context in Security Investigator with reference times appeared first on Dynatrace news.

Dynatrace introduces the next great addition to collaborative features for security investigators. You can now duplicate cases in Security Investigator.

Why is duplicating a case document important? Imagine having a case from last month that you need to reuse again. You can now create an exact copy of the existing case in seconds, saving you the hassle of starting from scratch. This promotes consistency, reduces errors, and streamlines your workflow.

Use cases for case duplication

Here are several additional use cases where duplicating cases can come in handy:

• Duplicate an existing case for sharing: You’re in the middle of an investigation, and your colleague asks if they can assist you with a case. You can duplicate the case and share the duplicate with your coworker so you don’t lock each other out.
• Create a snapshot of a current case: You’ve reached a critical point in an investigation, and to prevent accidental data loss, you want to create a quick snapshot of the investigation. So, you duplicate it and save a version of the case as a backup.
• Duplicate a case shared with you: If a case is shared with you in read-only mode, locked by someone else, or you simply want to avoid meddling with another person’s case, you can duplicate it and continue working on it.

Duplicating cases isn’t just about making copies; it’s about empowering you to work smarter. With a single click, you can duplicate any case, customize it as needed, and focus on what matters most. It’s perfect for creating new cases or templates of existing cases easily and precisely.

With our recently announced Security Investigator templates, you can create templates from any shared case.

For example, a coworker shares a case with you that, following some minor adjustments can serve as a recurring investigation. You can duplicate such a case, edit it, and create a template for later use.

Get started

You can try out the new case-duplication functionality now on the Dynatrace Playground.

The post Duplicate cases: A game-changer for Security Investigator productivity and efficiency appeared first on Dynatrace news.

We’re proud to announce that Dynatrace has introduced a new capability to speed up your security incident response and root cause analysis use cases with case templates in Security Investigator. Case templates allow you to start your investigations faster using prepared queries and evidence as a boilerplate.

Repetitive tasks in security incident responses waste time

When investigating incidents in production, engineers typically start each investigation with similar queries to understand what happened and where to look next, though the specifics can vary. Familiarity with the production environment’s artifacts (for example, environment names and deployment labels) is crucial, but gathering this information can be time-consuming, creating overhead before the actual investigation begins. The accumulated time spent on such activities before starting incident resolution can be a lot of overhead for engineers and the company.

Case templates assist in kicking off investigations

Dynatrace introduces case templates in Security Investigator to speed up new investigations and save engineers from manual, repetitive work.

Case templates provide engineers with a boilerplate for their investigation. They offer ready-to-be-executed DQL queries and the required artifacts about your environment as evidence lists, saving time on manual query creation or copying the queries from incident response playbooks.

Craft templates in Security Investigator

Case templates can be created from existing investigations or downloaded from other sources, like Dynatrace blog posts or documented use cases. You can select any current case and create a template from it.

Once a template is created, you can adjust it via the Template Editor to better suit your needs.

Case templates can be downloaded from your environments for safekeeping and uploaded for use in different environments. They can be shared directly with other Dynatrace users or made available to everyone on the same tenant. You can download and upload templates from your tenant to facilitate wider adoption of templates in multi-tenant environments or share templates with others in the community.

You can learn more about case templates in Dynatrace documentation.

The post Reduce security incident response time with case templates appeared first on Dynatrace news.

You’re in the middle of threat-hunting activities using Dynatrace and discover that some of your assets are trying to resolve a suspicious DNS name to extract data via covert channels like DNS tunneling. You now want to detect such events automatically by creating a custom Dynatrace security event. This blog post explains how to achieve this goal.

Following the Route53 log use case in Dynatrace threat-hunting documentation, a DNS query record is structured in the following way:

{
    "version": "1.100000",
    "account_id": "684399999995",
    "region": "us-east-1",
    "vpc_id": "vpc-026e4d09999995",
    "query_timestamp": "2024-08-17T07:24:57Z",
    "query_name": "250607.56755a335668636d52634969497343694167496e4a6c59584e766269493649.434a4762334a696157526b5a5734694c416f6749434a6b5a5852686157787a.tiitha-maliciousdomain.com.",
    "query_type": "A",
    "query_class": "IN",
    "rcode": "NOERROR",
    "answers": [
        {
            "Rdata": "34.667.67.208",
            "Type": "A",
            "Class": "IN"
        }
    ],
    "srcaddr": "172.31.1.6",
    "srcport": "42241",
    "transport": "UDP",
    "srcids": {
        "instance": "i-0b56150a99995e6d",
        "resolver_endpoint": "rslvr-out-c8e7c49999999fb5a"
    }
}

Let’s say that the meaningful fields you need for the security event are:

• timestamp when the DNS query was executed (query_timestamp);
• source IP where the query was performed (srcaddr);
• DNS query requested (query_name).

The JSON matcher, with its ability to parse out selected members, allows you to use the following simple DQL expression to extract only the fields you need:

json{
  timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp,
  string:query_name,
  ipaddr:srcaddr
}(flat=true)

As a result, you now have three additional fields with their respective types and values.

To add more accuracy to the query you’re about to automate, you need to extract the domain portion from the query_name field. This can be achieved with the following DQL expression:

LD* '.'? ( (LD '.' LD):domain '.' EOS)

That DQL pattern translates to something like this:

“Please extract two line-data (LD) portions from the content separated by a dot LD '.' LD) that is immediately followed by a single dot ('.') and then reaches the end of the string (EOS). A single dot and some line data might precede it.”

So, the pattern will match the highlighted DNS names below:

Adding a filter for the correct domain and limiting the query to only the last 5 minutes gives you the following DQL query to automate:

fetch logs, from: -5m
| filter aws.log_group == "/aws/route53/unguard-secla-demo/resolver-logs"
| parse content, "json{timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp, string:query_name, ipaddr:srcaddr}(flat=true)"
| parse query_name, "ld* '.'? ( (ld '.' ld):domain '.' eos) "
| filter domain == "tiitha-maliciousdomain.com"
| fields query_timestamp, srcaddr, query_name

Automate the query

Dynatrace AutomationEngine provides the ability to automate such periodic query executions. When you’re satisfied with your DQL query and are ready to automate it, select the Open with button in the lower right-hand corner of the query window.

As a result, a modal dialog will display, providing you with different possibilities about where to open the DQL query. In the current use case, you should open the query in Workflows.

Modify the default workflow

To execute the workflow automatically every 5 minutes, change the default on-demand trigger to either a cron or time-interval trigger. You can easily achieve this by selecting the trigger box in Workflows and following the instructions in the right-hand pane. You can read more about workflow triggers in Workflow schedule trigger documentation

Ingest query results as security events

The simplest way to do this is to use Dynatrace OpenPipeline. OpenPipeline allows you to create custom endpoints for data ingestion and process the events in the pipeline (for example, adding custom pipe-dependent fields to simplify data analysis in a later phase).

Set up a custom pipeline

The best way to set up a security event ingestion to Dynatrace is via Dynatrace OpenPipeline. You can create your own ingest endpoint that will then perform custom processing activities, create additional fields, store these events in separate buckets, and much more. For the current use case, we can create a custom security event pipeline to which we can send all the automated security events we have created from the Security Investigator, add respective fields for future analysis, and store them in the default security events bucket.

Now, create a general ingest endpoint for this use case and any automated detections we want to initiate from the Security Investigator in the future.

1. Go to OpenPipeline.
2. Choose Events > Security Events.
3. On the Ingest Sources tab, select + Source.
4. Give your pipeline a name, for example, Automated discovery from Security Investigator.
5. Choose a path to the new ingest source, for example, Signals.
6. Since we currently don’t want to create any dynamic rules to our pipeline, leave Routing set to static and choose the Custom events pipeline as the route.
7. Go to the Pre-processing tab and add a new processor with the type Add Fields.
8. Give the processor a name, for example, My custom processor.
9. Add the fields that you want to be automatically added when an event passes the pipeline. In our case, we will add the following fields:
   – kind == “SECURITY_EVENT”
   – provider == “Automated Detection Script”
   – category == “AUTOMATED_DETECTION”
   – provider_product == “Security Investigator App”
10. To figure out which fields to add for custom security events, refer to the semantic dictionary.
    Following the overall convention here will simplify analyzing this data later, together with other security events created by other sources.
11. You can test the processor with sample data to see what kind of event will be created based on the processor, and if you’re happy with the result, select Save.

For the current demo, this is enough, however, there are lots of things that you could configure additionally. Take a look at the Ingest custom security events via API documentation to learn more about sending security events to OpenPipeline.

Create and secure the token for event ingest

To create the token needed to ingest events, follow the steps in How to ingest data (events) documentation. When you have created the token, we recommend that you protect it in the Credential Vault instead of just using it in plain text in the automation engine.

1. Open Credential Vault.
2. Select Add new credential.
3. Select token as the credential type and give it a meaningful name.
4. Select App Engine as the credential scope.
5. Paste your ingest token into the token field and select Save.

After saving the credentials, an ID is presented to you in the form of CREDENTIALS_VAULT-12345. Write this down for later usage.

Ingest records through the OpenPipeline custom endpoint

Let’s return to our workflow and add a follow-up task for the DQL query. Choose the JavaScript task type and start writing the code to ingest security events.

Consider the following code for your function:

import { execution } from '@dynatrace-sdk/automation-utils';
import { credentialVaultClient } from '@dynatrace-sdk/client-classic-environment-v2';

export default async function ({ execution_id }) {

  const ex = await execution(execution_id);

  // load the token from the credential vault by its' ID
  const token = await credentialVaultClient.getCredentialsDetails({
    id: "CREDENTIALS_VAULT-F12345",
  }).then((credentials)=> credentials.token);

  // loop through the DQL query results and ingest them one by one
  var result = await ex.result('execute_dql_query_1');
  result.records.forEach (record => {
    ingest_event(token, record);
  });
}

async function ingest_event(token, payload) {

  payload["event.description"] = "Suspicious DNS query from AWS DNS Logs";

  fetch('https://<dynatrace_env>.dynatrace.com/platform/ingest/custom/events.security/signals', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json; charset=utf-8",
      "Authorization": `Api-Token ${token}`
    },   
    body: JSON.stringify(payload)
  }).then(function(response) {
    console.log(response);
  })
}

The code loads the token from the Credential Vault that will be used to ingest the events to OpenPipeline. Then, we iterate through the result set we got from the execute_dql_query_1 task and pass each event along with the ingest token to the ingest function, which will POST the event to the OpenPipeline ingest endpoint that we created earlier. The response to the request is shown in the console for debugging purposes.

We’re also adding a custom field called event.description to the event. You can add custom fields to your event from the OpenPipeline processing as well, but since we want to keep the ingest source as general as possible, it makes more sense to add the event-specific fields near the event, not the pipeline.

Now, give it a distinct name and test if it runs perfectly. Ingesting events to Grail via OpenPipeline should result in a status: 202 that is visible in the run_javascript_1 log console.

Fetch the security events from Grail

To view the newly created security events, follow the same principles as with fetching any other security data from Grail. Using the following DQL query, you can see all the events you have started to ingest:

fetch events
| filter event.kind == "SECURITY_EVENT"
| filter event.category == "AUTOMATED_DETECTION"

To feel the power of the semantic dictionary and to illustrate what you gain from using it, consider the following query. If you want to visualize all the security events ingested to Grail over the last 12 hours by event.category, it is quite simple with the following query thanks to the unified field names:

fetch events, from: -12h
| filter event.kind == "SECURITY_EVENT"
| makeTimeseries count(), by: { event.category }, interval:1h

The result could be a line chart like this:

Of course, you can follow your own naming convention, but this would make it much harder to view all the security events (both custom and Dynatrace-created events) in conjunction.

Operationalize the results

You can create additional workflows for your security events in Grail. For example, you can create a workflow that notifies you in Slack when a new event is created or integrate your workflows with Jira to create a ticket every time such events arise.

You can create a custom integration to your own SIEM system (similarly as we did for the OpenPipeline) for events that can enrich your security data in other environments. Consider your use case and work processes to determine the best way to approach security events available in Grail.

The post Generate security events from Dynatrace Security Investigator via OpenPipeline appeared first on Dynatrace news.

When kicking off a threat hunting activity, you can immediately share your investigation with your teammates, to keep them up-to-date and allow them to collaborate with the ongoing investigation.

Sharing a case

There are several ways to share a case:

• Personal or group sharing – Select colleagues by name or a group to share the case.
• Link sharing – Generate and distribute a shared link in Slack or include it in your report.

Both share modes support either read-only or edit privileges. Read-only mode allows you to browse the case. Edit mode allows you to execute queries and modify the case and its contents.

You can also combine the modes: You can create a read-only link to distribute the case in the organization for everyone to view and give edit privileges to your teammates for collaborative investigations at the same time!

Handle shared cases

You can identify shared cases by a small Shared icon on the main page of Security Investigator. Cases you have shared are marked with a blue icon; cases shared with you include a white icon. Hover over the Shared icon for more details about who shared the case with you. If the case is shared with you in read-only mode, you’ll see the respective label next to the icon.

Empower your investigations with joint editing

When you grant your teammates edit access, they can perform any investigative action, from executing queries and creating new branches to modifying the query tree or removing evidence from the case.

To avoid integrity issues, only one editor is allowed at a time. When an investigator opens a case for editing, it’s read-only for all other investigators, even if they have edit privileges or are the case owners; other investigators see a notification stating that the case has been locked and who is currently dealing with it.

Control access permissions

When you grant your teammates edit access, they can perform investigative actions within the case, but the ownership remains the same. Only the owner can grant and revoke sharing permissions or delete the case.

If the shared link has been distributed too widely or leaked, or the owner wants to revoke link access to the case, you can always remove or recreate it. The old link will no longer provide access to the case.

What’s next

• Read more about the amazing features of Security Investigator in Dynatrace Documentation.
• Check out threat-hunting use cases with Dynatrace Security Investigator.

The post Collaborate with peers in hunting security threats appeared first on Dynatrace news.

What is CVE-2024-6387 (regreSSHion vulnerability)?

The regreSSHion vulnerability (CVE-2024-6387) has been discovered by Qualys Threat Research Unit (TRU) as a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH server (sshd) in glibc-based Linux systems. The vulnerability, a signal handler race condition in OpenSSH server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based Linux systems, which is a significant security risk. This race condition affects sshd in its default configuration.

Technical Overview

• Name: CVE-2024-6387
• Description: A security regression (CVE-2006-5051) was discovered in OpenSSH’s server (sshd). There is a race condition which can lead sshd to handle some signals in an unsafe manner. An unauthenticated, remote attacker may be able to trigger it by failing to authenticate within a set time period.
• CVSS Base Score: 8.1 HIGH
• Source: CVE (at NVD; CERT, LWN, oss-sec, fulldisc, Red Hat, Ubuntu, Gentoo, SUSE bugzilla/CVE, GitHub advisories/code/issues, web search, more

What is OpenSSH?

OpenSSH is an essential tool for secure communication and administration in Unix-like operating systems and is also available for Windows. It is part of the broader OpenBSD project and is widely adopted due to its robustness and security features.

What is the potential impact of regreSSHion vulnerability?

Exploiting this vulnerability can lead to full system compromise, where an attacker can execute any commands or code with the highest privileges. This can result in a complete system takeover, malware installation, data manipulation, and the creation of backdoors for persistent access. Moreover, gaining root access will enable attackers to bypass critical security mechanisms such as firewalls, intrusion detection systems, and logging mechanisms, further obscuring their activities. This can result in significant data breaches and leakage, giving attackers access to all data stored on the system, including sensitive or proprietary information that can be stolen or publicly disclosed.

To avoid sounding too alarming, we have to emphasize that this vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. As written in the Qualys white paper, “In our experiments, it takes ~10,000 tries on average to win this race condition. ” To our knowledge as of writing this article, no properly working exploits have been created.

How to mitigate the risk of CVE-2024-6387

The following versions of OpenSSH are impacted by this vulnerability:

• Versions earlier than 4.4p1 (unless patched for CVE-2006-5051 and CVE-2008-4109)
• Versions from 8.5p1 up to, but not including, 9.8p1

OpenBSD systems are unaffected by this, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability.

There are three strategic paths to take to mitigate this threat:

• Patch Management: Quickly apply available patches for OpenSSH or upgrade to the relevant version that is not impacted by this vulnerability
• Access Control: Limit SSH access via network-based restrictions
• Network Segmentation: Divide networks to restrict unauthorized access and lateral movements from other systems

How to detect regreSSHion exploitation attempts with Dynatrace

This vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. Thanks to this noisy approach it can be detected from your sshd logs with Dynatrace DQL queries using Dynatrace Security Investigator.

Look for timeout events

Exploitation attempts for this vulnerability can be identified by many lines of “Timeout before authentication” in the logs. The log line in the logs could look something like this:

Jul  1 09:40:39 [localhost] sshd[787404]: fatal: Timeout before authentication for 192.168.12.45 port 17296

To search for such events from your sshd authentication logs stored in Grail, the following DQL query can be used:

fetch logs, from: -48h
| filter contains(content, "Timeout before authentication")

If you don’t get any results, that might be good news! You can further expand the timeframe of your query by changing the from time to -7d. This will get you logs from last week to make sure that no one tried to exploit the system during the last week.

Look for a high volume of log entries

There’s a chance that someone has tried to attack your system with a different attack or that the attack did not succeed as intended. For example, some Proof-of-concept attacks have failed, and these failures write various error messages to the victims’ sshd logs. It’s always good to check the log trend line in general to detect such attacks and when hunting for the unknown.

It’s easy to create a timeseries from your logs using DQL. With the following query, you can create a chart that shows sshd log line count from the previous day, grouped by host and aggregated at an hourly interval.

fetch logs, from: -24h
| filter contains(content, "sshd[")
| makeTimeseries count(), by: { dt.entity.host }, interval:1h

From the chart, it’s easy to see that nothing anomalous was detected using this query: no weird peaks or anomalies could be extracted from the chart.

Look for suspicious IP addresses in sshd logs

To ensure no one has tried to access your services from a suspicious location, it’s valuable to extract all the IP addresses from your sshd logs and see if something suspicious emerges from there. To do that, the following DQL query is helpful:

fetch logs, from: -2d
| filter contains(content, "sshd[")
| parse content, "ARRAY{ ld* ipaddr:ip }{0,10}:source_ip"
| expand source_ip
| summarize count(), by:source_ip
| sort source_ip desc

This query will fetch your sshd logs from the past two days, will extract ALL the IP addresses (both ipv4 and ipv6) from the content and will sort the result based on their occurrances to your results. If you discover any suspicious IP addresses from this list, you can start your investigation based on the discovered threat! If you see only your own addresses, you can rest assured: your sshd has not been accessed by an unknown IP address.

Analyze network flow logs

Last but not least, your network logs are the ultimate source of data. If you’re using AWS for your cloud platform and you have forwarded your VPC flow logs to Dynatrace, you can use DQL to look for your potential victims AND suspicious IP addresses that have tried to access your servers using SSH.

Consider the following query:

fetch logs, from: -24h
| filter aws.log_group == "/aws/vpc/flow_logs"
| parse content, """INT:version ' '
NSPACE:account_id ' '
NSPACE:interface_id ' '
('-' | IPADDR:src_addr) ' '
('-' | IPADDR:dst_addr) ' '
('-' | INT:src_port) ' '
('-' | INT:dst_port) ' '
('-' | INT:protocol) ' '
('-' | LONG:packets) ' '
('-' | LONG:bytes) ' '
TIMESTAMP('s'):start ' '
TIMESTAMP('s'):end ' '
LD:action ' '
LD:log_status"""
| filter dst_port == 22

The query will fetch all the Grail logs ingested from the CloudWatch log group called /aws/vpc/flow_logs. Using the VPC flow log default pattern available in DPL Architect, we can extract the meaningful fields to see only the network traffic targeting the SSH port.

From this dataset, we can create metrics based on the destination IP address to detect all the potential victims by adding the following code snippet to the previous query:

| filter action == "ACCEPT"
| makeTimeseries count(), by: {dst_addr}, interval: 1min

Viewing the results as a graph, you can see that some servers have received an anomalous amount of SSH traffic towards them! From here, you can continue to look further into the specific destinations to understand if this is malicious activity.

To understand who is targeting you, a similar query can be used. Instead of dst_addr use a src_addr to see the most active sources that have been targeting any of your ssh daemons on all the servers in the respective VPC.

| filter action == "ACCEPT"
| makeTimeseries count(), by: {src_addr}, interval: 1min

The results show that of the 725 SSH sources, three stand out with a huge load of requests that require further analysis.

Next steps for mitigating regreSSHion vulnerability

This particular vulnerability is challenging to exploit due to its race condition nature. However, all vulnerabilities should be taken seriously, their potential impact on your environment analyzed, and, where possible, mitigated. If you plan not to take any action for whatever reason, you should at least analyze the potential risk and make an informed decision before taking any action.

The post RegreSSHion vulnerability: Detecting CVE-2024-6387 in OpenSSH appeared first on Dynatrace news.

The growing complexity of modern multicloud environments has created a pressing need to converge observability and security analytics.

Security analytics is a discipline within IT security that focuses on proactive threat prevention using data analysis. As attackers become more skilled, threats can become more detrimental to organizations if they go undetected. A proactive approach to application security is essential: by constantly collecting and analyzing data, security analytics enables teams to catch problems before they escalate.

In the past five years, delivering innovation more efficiently has remained at the forefront of customer demand. As environments began to scale with cloud-native and microservices architectures to meet this demand, security approaches didn’t evolve accordingly. The result: Environments are more vulnerable to threats and more difficult to secure. In fact, according to a recent survey, nearly 70% of chief information security officers agreed that vulnerability management has become more difficult as the complexity of their software supply chain and cloud ecosystem has increased.

The ripple effect of increased risk compounds the problem. With more alerts and greater difficulty identifying false positives, security teams experience frustration and burnout. A lack of common tooling, common language, and collaboration further inhibits productivity and prolongs remediation.

At the 2024 Dynatrace Perform conference in Las Vegas, Gerhard Byrne, Dynatrace principal product manager, and Susan St. Clair, principal security solutions engineer, discussed how organizations can take a more proactive approach to threat detection and incident resolution.

During their breakout session, Byrne and St. Clair demonstrated how the Dynatrace platform accelerates remediation by enriching security data with observability context and actionable insights to protect environments against exploitation or lateral movement.

Threat hunting expectations vs. reality

In a perfect world, threat hunting and incident resolution would be a linear, straightforward process. Ideally, after fetching data and filtering, an organization could enrich the findings with observability data to get better insights into the nature of the alert. With these insights, the team could identify the threat and understand the nature of the incident. This allows them to react accordingly and return the system to a secure state.

But with the complexity of modern cloud environments — including the associated software supply chains and siloed toolchains — and the increasing sophistication of today’s attackers, threat hunting is unpredictable. In reality, security teams aren’t aware of all the unknown unknowns in their environments. After fetching, filtering, and enriching information with observability data, security teams might change their hypothesis about what’s occurring. This revision of assumptions might happen multiple times, causing engineers to lose track of previous hypotheses, patterns, and evidence. Keeping threats documented is a challenge: Engineers typically open numerous tabs to maintain context, which is tedious and can create error. Remediating a vulnerability can thus take far longer than anticipated, which can be detrimental when the risk is high.

“As defenders, we need to embrace different paths and possibilities like our adversaries are doing today,” Byrne said. “Just going down a checklist will not help you find new threats.”

Streamline threat hunting and accelerate resolution with Dynatrace Security Investigator

During the conference, Dynatrace announced the new Security Investigator app on the platform. The app enables security teams to investigate threats faster, obtain accurate and observability-enriched results, and maintain context throughout the weaving paths on which security investigations might lead.

Security Investigator demo

St. Clair began her demonstration of the app with the following scenario: She receives a Slack alert that an anomaly was detected and there has been unauthorized access to a Kubernetes cluster monitored via OneAgent.

To begin, St. Clair filtered the alert for a time window of a couple of hours to focus her analysis. Using Dynatrace Query Language in Grail, St. Clair determined what log data was available to her. With each execution, data appears in a query tree. “As I’m building out this investigation, each of these nodes is being created for me automatically,” she said. “As part of that documentation, I can easily go back and forth to see what was executed.”

St. Clair then used the Dynatrace Pattern Language (DPL) to make the data more usable. She used the DPL Architect to create her own patterns in addition to the platform’s out-of-the-box patterns for parsing the data. After running an audit log pattern, she wanted to understand why she received the Slack alert and which object the attacker had accessed so she could then focus on the unauthorized responses. Running this query, suspicious IPs arose, and she saved them as evidence in a new folder she created within the app.

As she continued to execute queries, St. Clair’s hypothesis began to change. She saw substantial traffic in a specific port, which was not necessarily malicious, but it was abnormal enough to warrant additional investigation. Fortunately, the query tree automatically creates new “branches” to support changing hypotheses and help engineers keep track of evidence and patterns. The query tree thus frees security professionals from tedious manual documentation, allowing them to focus entirely on finding the unknown unknown.

Finally, St. Clair found running malware on the system, pivoting the investigation from the initial authentication alert. By the end of an investigation, she had a visual representation of the process from start to finish.

“I can keep track of where I went. [The data is] documented, shareable, collaborative, and available for further investigation,” St. Clair said.

The road ahead: enriched security analytics with Dynatrace

Organizations can benefit most from their security investigations using the abundant data in the Dynatrace platform. The platform’s broad and deep observability identifies where problems initiate as well as their dependencies using PurePath. Teams can use Real User Monitoring and Session Replay to track user-facing activity in real time. This helps them understand how an attacker accessed an application, how they interacted with it, how traces originated, and more. Teams can also create management reports and use Dynatrace Notebooks to easily share their security investigation data with their peers.

“Security is a team sport,” Byrne said. “The next time you’re in a war room, you can be the person who provides insights and conclusions based on the wealth of data that is available at your fingertips with Dynatrace platform.”

The post Advanced security analytics to resolve incidents quickly and streamline threat hunting appeared first on Dynatrace news.

As a security analyst investigating security incidents or threat hunting, you often must navigate among multiple executed queries and their respective results (in other words, investigation steps). You also need to manage the evidence you gather during investigations and reuse it when building additional queries. All this must be as fast and seamless while maintaining the investigation context.

Navigating investigations and threat hunting can involve saving executed queries and found evidence in unstructured text documents that can quickly lose their context. Manually documenting investigation steps can lead to human errors, like forgotten or wrongly documented queries or evidence, which will result in missed threats and wasted time. As the investigation grows and the different queries accumulate, it becomes harder to maintain the throughlines among the different branches.

The Dynatrace Security Investigator app provides branching navigation, making it easy to track your path through investigations.

Navigate in any direction within your investigation

Investigations in Grail are conducted using the Dynatrace Query Language (DQL). For multiple investigation steps, the Security Investigator provides a representation of the steps in a tree-like view. With this visualization, you can keep track of your investigation and navigate to previous steps and respective results in the investigation history—all in context, with no need to re-execute anything.

Every new DQL query execution creates a new descending node while keeping the integrity and hierarchy of previous queries and results unchanged.

Easily track threat-hunting twists and turns

Threat hunting is a nonlinear process. When looking for the unknown, you often stumble upon an unexpected thread of evidence that sparks a new idea and takes the investigation in a new direction. To support this nonlinear approach, we’ve created the investigation branching feature. When modifying and re-executing a previous step, the app creates a new branch while keeping the original investigation path intact and visible in the query tree.

You can create any new branches for new ideas and hypotheses while retaining the ability to navigate back to your initial train of thought. With the ability to customize and name branches, you can simplify locating historical steps without having to click through them individually.

The final visual representation provides a comprehensive overview of the entire investigation. You can see in a structured way how you reached your conclusions without needing to manually document the investigation flow.

With Security Investigator, you can always find the last working query, which was easy to lose in the past.

Character precision on a petabyte scale

Security Investigator increases the speed of investigation flows and the precision of evidence, leading to higher efficiency and faster results. Both outcomes are game-changers in time-critical situations, where answers are expected “yesterday.”

View raw content details when you need them

Data ingested into Grail is kept in its original format. A schema is applied only during query execution, or as we call it, “schema-on-read.” This makes data immune to format changes and enables you to choose your data structure based on your use cases.

The Security Investigator app enables you to view all data in its original format, regardless of the content, including characters that other tools might hide or not interpret. Enabling the multi-line content details view shows the stack traces with their line breaks in their original form, enabling you to understand the data much faster.

Find the needle in the “needle stack”

Finding a small metallic object in a stack of non-magnetic objects is relatively easy. Finding a particular object among millions of similar objects is trickier. With Security Investigator’s flexible filtering, you can create accurate DQL query filters quickly and efficiently.

If you need to fetch raw content from Grail and filter the results based on a selection of the content, you don’t need to copy partial records and manually paste them into an appropriate filter. Instead, you can easily select the portion of the text string you need and add it to your DQL query directly from the context menu.

To add multiple values to a filter, you can either select individual values or a range of values and apply them as a single filter in your DQL query. Only the unique values from the selection are added to the query, so you don’t have to investigate unneeded values.

Use your findings to enrich your investigation

With Security Investigator evidence management, you can persist relevant findings by attaching them to your investigation. Whether it is a suspicious IP address or a potential indicator of compromise, you can attach it to the investigation and use it later.

You can create type-specific filters to enable fast and accurate results, either focusing only on suspicious activities or filtering out the safe events that don’t require your attention.

Continue hunting for the unknown

Security use cases such as threat hunting and incident analysis are nonlinear, evidence-driven activities in which you don’t know what you’re looking for. You’re not pursuing a specific target; rather, you’re looking for the unknown. With Security Investigator, Dynatrace enables you to be more efficient, investigate faster, and be more precise.

Use Dynatrace Security Investigator for faster and more precise evidence-based security investigations.

The post Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator appeared first on Dynatrace news.