Tiit Hallas | Dynatrace news https://www.dynatrace.com/news/blog/author/tiit-hallas/ The tech industry is moving fast and our customers are as well. Stay up-to-date with the latest trends, best practices, thought leadership, and our solution's biweekly feature releases. Fri, 16 Jan 2026 07:55:01 +0000 en hourly 1 From incident response to everyday analytics: Introducing Dynatrace Investigations https://www.dynatrace.com/news/blog/from-incident-response-to-everyday-analytics-introducing-dynatrace-investigations/ https://www.dynatrace.com/news/blog/from-incident-response-to-everyday-analytics-introducing-dynatrace-investigations/#respond Wed, 07 Jan 2026 18:22:44 +0000 https://www.dynatrace.com/news/?p=72376 Query tree filter

Unlocking actionable insights from data is no longer just a security concern—it’s essential for any team that needs to make sense of large, complex data sets. The recently renamed “Investigations” app (previously known as “Security Investigator”) empowers you to analyze logs, events, metrics, and traces using DQL, transforming raw information into actionable insights that drive business outcomes.

The post From incident response to everyday analytics: Introducing Dynatrace Investigations appeared first on Dynatrace news.

]]>
Query tree filter

Transform how you derive actionable insights from data

Have you ever struggled to make sense of a massive data set or turn raw data into actionable business insights? Investigations handles all this for you—and more—at the scale modern organizations demand.

Whether you’re troubleshooting API call throttling, debugging AWS integration issues, or accelerating root cause analysis, the Investigations app provides a flexible toolkit for exploring your data, allowing you to:

  • Analyze large DQL results in their original form at a detailed level
  • Maintain your entire investigation flow and historical queries in context via the query tree
  • Perform complex investigations on data stored in Dynatrace Grail®
  • Build DQL queries quickly and efficiently based on your findings
  • Save and reuse evidence to refine queries and uncover answers
  • Pivot your queries based on the metainformation attached to log records
  • Analyze the observability metrics connected to your log sources
Figure 1. Derive business insights from data in Grail with Investigations
Figure 1. Derive business insights from data in Grail with Investigations

With Investigations, you can easily navigate through investigation history to review queries and results—streamlining incident response, accelerating root cause analysis, and delivering deeper contextual insights from your data.

Get started with practical use cases

As strong advocates of “learning by doing,” we understand you may be curious about how to start using the Investigations app more broadly. You don’t need to start from scratch—there are plenty of documented scenarios you can customize for your own purposes. To help you get started, explore these informative tutorials:

If threat hunting is more your style, you can start with these tutorial-style investigation challenges:

There are, of course, many more use cases to explore—Investigations can be used with any data in Grail: logs, events, metrics, traces, and even performance data.

Drive insights across all your data

Since its launch, the recently renamed Investigations app has been key to unlocking insights in Grail, allowing teams to analyze metrics alongside logs and connect traces during incident response—this is a game-changer for both security and observability. Over time, the capabilities of the Investigations app were extended well beyond security, supporting a wide range of analytical scenarios.

Despite the name change, all the app’s existing features, including application IDs, which are used in intents and in the browser address bar, remain unchanged, so no updates to integrations or bookmarks are needed. And the roadmap will continue to prioritize incident response and accelerations of investigations.

For ease of use, Investigations will remain listed under “Security” on the Dynatrace platform and in Dynatrace Hub.

For complete details, please see Investigations documentation.

Figure 2. Use cases in the Investigations app
Figure 2. Use cases in the Investigations app

Ready to get started?

Investigations is a built-in app available in all SaaS environments beginning with Dynatrace SaaS version 1.330. Best of all, this app requires no additional subscriptions or privileges—so you can start using it immediately without waiting for access permissions and unlock deeper insights for your business.

The post From incident response to everyday analytics: Introducing Dynatrace Investigations appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/from-incident-response-to-everyday-analytics-introducing-dynatrace-investigations/feed/ 0
Enhanced incident response based on performance-metric insights https://www.dynatrace.com/news/blog/enhanced-incident-response-based-on-performance-metric-insights/ https://www.dynatrace.com/news/blog/enhanced-incident-response-based-on-performance-metric-insights/#respond Thu, 07 Aug 2025 15:54:40 +0000 https://www.dynatrace.com/news/?p=70345 Query tree filter

Solving incidents or finding root causes is a time-critical activity that requires logged evidence to understand what really happened in a system. Whether to prevent such incidents from happening again or to rule out a malicious hacking attempt, getting answers is the key, and logs are your best source for getting such evidence. However, analyzing […]

The post Enhanced incident response based on performance-metric insights appeared first on Dynatrace news.

]]>
Query tree filter

Solving incidents or finding root causes is a time-critical activity that requires logged evidence to understand what really happened in a system. Whether to prevent such incidents from happening again or to rule out a malicious hacking attempt, getting answers is the key, and logs are your best source for getting such evidence.

However, analyzing only logs might not be enough. For example, if the incident resulted in a system crash, it would be important to also see the utilization of the system just before the incident occurred or to understand what happened in the system around utilization peaks.

This is why Dynatrace has introduced a new advanced investigation feature for Security Investigator that gives you faster insights into Performance Metrics.

Solving latency issues with metric insights

Dynatrace Security Investigator is one of the built-in apps that ships with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into the Dynatrace Grail® data lakehouse.

Imagine you’re performing root cause analysis on a long-running query using Security Investigator. You’ve found the logs for the request that was running for a long time, and you want to understand what the CPU utilization of the application was at the time of the request.

By simply right-clicking on the log record in your results, you can choose either the relevant pod, container, host, or any other dimension available in the log record and choose CPU utilization.

Security Investigator dashboard in Dynatrace screenshot

As a result, you can see the CPU utilization chart from the specific pod around the time of the log record in the context of your investigation next to your logs. You can modify the aggregation function or change the visible metric for the pod, if needed. CPU utilization chart in Dynatrace screenshot

The red indicator on the chart shows the selected log record’s timestamp from your results table. If you choose any of the other log records in the results table, the red indicator will change its position on the chart, enabling you to update the CPU utilization chart with your log-analysis context.

Persisted throughout the case

When navigating to other query nodes in your query tree, you can still see the CPU utilization chart, enabling you to analyse other logs in the same context. For example, you might want to analyze your network flow logs from the same period and see how some network requests from the same time might be connected to the pod’s CPU usage. Selecting network flow events moves the red indicator’s position on the chart (shown as a dashed line due to different entity values).

Security Investigator dashboard in Dynatrace screenshot

Get started

Visit the Dynatrace Playground to see performance metrics in action or learn more about how performance metrics add relevant context to your security investigations in Dynatrace Documentation.

The post Enhanced incident response based on performance-metric insights appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/enhanced-incident-response-based-on-performance-metric-insights/feed/ 0
Security Investigator offers reputation analysis and context for IP addresses https://www.dynatrace.com/news/blog/security-investigator-offers-reputation-analysis-and-context-for-ip-addresses/ https://www.dynatrace.com/news/blog/security-investigator-offers-reputation-analysis-and-context-for-ip-addresses/#respond Mon, 14 Jul 2025 19:08:35 +0000 https://www.dynatrace.com/news/?p=69976 Query tree filter

Resolving incidents and finding root causes are time-critical activities that require logged evidence to understand what happened in the system. Whether to prevent such incidents from happening again or to rule out a malicious hacking attempt, getting answers is the key, and logs are your best source for such evidence.

The post Security Investigator offers reputation analysis and context for IP addresses appeared first on Dynatrace news.

]]>
Query tree filter

Log data alone, however, is often not enough to give you a complete overview. In the case of security incidents, the origin of a failed request or a login attempt is often the most valuable information you have in triaging a potential incident and determining its severity.

To speed up your threat hunting and incident response activities, Security Investigator now allows you to enrich IP addresses with additional context from third-party threat detection databases.

Fast insights into IP addresses

Dynatrace Security Investigator is one of the built-in apps shipped with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into the Dynatrace Grail® data Lakehouse.

Imagine you’re solving a security incident and need to understand more about the origin of an attack. You want to check the IP address reputation analysis provided by AbuseIPDB or VirusTotal, but manually looking up and pasting these addresses takes too much time. Not to mention, you want to keep this external reputation information tied to your investigation for future access.

IP address reputation analysis

By right-clicking an IP address in Security Investigator and selecting Enrich IP, you can choose your configured enrichment source to enrich the address. For this first release, Dynatrace offers IP enrichment insights from AbuseIPDB and VirusTotal.

Suppose you find something suspicious in your log data related to a certain IP address. You can now add the IP address and additional reputation analysis to your Suspicious IP evidence list in Security Investigator.

Security Investigator

The IP address and corresponding enrichment data are now kept in the context of your investigation; you can revisit this enrichment data at any time.

IP address and corresponding enrichment data

Automatic IP evidence enrichment

You can now automatically enrich all the IP addresses you collect with evidence. In the evidence list enrichment menu, choose if enrichment should be enabled for all added elements by default, and which connection should be used to enrich the IP addresses that are added to your evidence list.

Automatic IP evidence enrichment

When you add new IP addresses to the evidence list, they’re automatically enriched on your behalf using your selected enrichment connection.

IP addresses to the evidence list

Select the details icon to quickly access the persisted IP enrichment data that’s fetched from each third-party source.

Visit the Dynatrace Playground to see IP enrichment in action or learn more about setting up IP enrichment in Dynatrace Documentation.

The post Security Investigator offers reputation analysis and context for IP addresses appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/security-investigator-offers-reputation-analysis-and-context-for-ip-addresses/feed/ 0
Pivot the perspective of your investigative queries with Security Investigator https://www.dynatrace.com/news/blog/pivot-the-perspective-of-your-investigative-queries-with-security-investigator/ https://www.dynatrace.com/news/blog/pivot-the-perspective-of-your-investigative-queries-with-security-investigator/#respond Thu, 19 Jun 2025 15:12:23 +0000 https://www.dynatrace.com/news/?p=69561 Query Tree filter

Dynatrace introduces a new advanced way to enhance your DQL queries in Security Investigator cases. The pivoting queries concept allows engineers to quickly change the investigation context by switching the scope of a query using available pivoting dimensions. Time is critical Imagine you’re investigating latency issues in your cloud applications by analyzing your Istio proxy […]

The post Pivot the perspective of your investigative queries with Security Investigator appeared first on Dynatrace news.

]]>
Query Tree filter

Dynatrace introduces a new advanced way to enhance your DQL queries in Security Investigator cases. The pivoting queries concept allows engineers to quickly change the investigation context by switching the scope of a query using available pivoting dimensions.

Time is critical

Imagine you’re investigating latency issues in your cloud applications by analyzing your Istio proxy logs with Security Investigator. You have hundreds of pods, each of which has application containers and Istio containers running side-by-side. You analyze all the Istio logs at once to get a comprehensive overview of the whole infrastructure, fetching logs for all long-running requests, and you want to investigate some of the requests to identify the issues behind the high latency.

Dynatrace Security Investigator is one of the built-in apps shipped with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into the Dynatracer Grail® data lakehouse.

Security Investigator allows you to:

  • Keep your whole investigation flow in context.
  • Perform complex security investigations on the data stored in Grail.
  • Build DQL queries based on your findings in a fast and easy way.
  • Save and use the found evidence to build your DQL queries and find answers to your questions.
  • Navigate with ease to any point in your investigation history and review queries and results.
  • Fetch detailed results in the original format to quickly understand the information.

Quickly shift your investigative perspective

Thanks to pivoting queries, it‘s now possible to quickly shift your investigation from one perspective to another. You can choose Pivot query by right-clicking any record in the results table in Security Investigator and selecting the dimension you want to pivot your query by, for example, Kubernetes pod. As a result, a new query node is created, containing all the logs from the same Kubernetes pod from which the Istio record originated, giving you the full context and all application logs from that pod.

Security Investigator in Dynatrace screenshot

Pivoting queries also work at scale: if you select multiple records and use pivoting queries with multiple values, multiple nodes will be created based on the distinct selected values. So, if you choose five result records and pivot query by trace_id, then five new query nodes will be created, which will contain results for each respective trace ID. This allows you to continue investigating each trace in your query branch, keeping the context and tracking your individual query history.

Query tree in Dynatrace screenshot

The pivoting dimensions are chosen from the metadata fields of your log records and can be further customized in Security Investigator. You can switch dimensions from, for example, a high-level Kubernetes cluster or host to the corresponding process group instance or cloud application, depending on your investigation’s purpose and the context you need.

The pivoting queries feature allows you to speed up your investigations and apply them in different contexts using the power of the query tree and other investigative features of Security Investigator.

Get started

Visit the Dynatrace Playground to see pivoting queries in action or learn more about how pivoting queries speed up incident response in Dynatrace Documentation.

The post Pivot the perspective of your investigative queries with Security Investigator appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/pivot-the-perspective-of-your-investigative-queries-with-security-investigator/feed/ 0
Create context in Security Investigator with reference times https://www.dynatrace.com/news/blog/create-context-in-security-investigator-with-reference-times/ https://www.dynatrace.com/news/blog/create-context-in-security-investigator-with-reference-times/#respond Tue, 27 May 2025 16:05:30 +0000 https://www.dynatrace.com/news/?p=69252 Query tree filter

Dynatrace introduces a new way to think about timestamps in your Security Investigator cases. The reference time concept uses relative timestamps to speed up incident response time. Time is critical Imagine an engineer performing root cause analysis by analyzing your application logs using Security Investigator. They discover the event in the logs that caused an […]

The post Create context in Security Investigator with reference times appeared first on Dynatrace news.

]]>
Query tree filter

Dynatrace introduces a new way to think about timestamps in your Security Investigator cases. The reference time concept uses relative timestamps to speed up incident response time.

Time is critical

Imagine an engineer performing root cause analysis by analyzing your application logs using Security Investigator. They discover the event in the logs that caused an incident, and as the next step, they want to analyze network flow logs from your Cloud Service Provider (CSP). Since network flow logs don’t contain any meaningful data about the request content or its response, it’s relatively difficult to understand which events in the network flow logs occurred before the incident and which ones followed it.

Understand which network flow log events occurred before and following an incident.
Figure 1. Understand which network flow log events occurred before and following an incident.

Dynatrace Security Investigator is one of the built-in apps shipped with Dynatrace. It’s designed for evidence-driven security use cases based on the logs, metrics, and traces ingested into Grail.

Security Investigator allows you to:

  • Keep your whole investigation flow in context.
  • Perform complex security investigations on the data stored in Grail.
  • Build DQL queries based on your findings in a fast and usable way.
  • Save and use found evidence to build your DQL queries and find answers to your questions.
  • Navigate with ease to any point in your investigation history and review queries and results.
  • Fetch detailed results in the original format to quickly understand the information.

Thanks to reference time, it‘s now possible to add the time perspective to keep track of the relative time between events you’re analyzing and the time that incidents occurred.

You can choose Set as reference time by right-clicking any Security Investigator timestamp field related to an event that caused an incident.

Select a timestamp in Security Investigator to set a relative reference time offset.
Figure 2. Select a timestamp in Security Investigator to set a relative reference time offset.

A new virtual column called timestamp_diff is then added to all the query nodes of the query tree. The virtual node contains the time difference between the set reference time and the timestamp of the event in the results table. So, even if you open another query node, you’ll still see the reference time offset field in the results menu displaying the time distance between the event and the set reference time.

Security Advisor displays relative reference time stamps on the right-hand side in the virtual offset column.
Figure 3. Security Advisor displays relative reference time stamps on the right-hand side in the virtual offset column.

The virtual offset column is created automatically for the first timestamp field in the results table (usually called timestamp), but it can be turned on for any timestamp field. You can do this from the column header menu or from the reference time menu.

Selecting the time offset columns in the Security Investigator app.
Figure 4. Selecting the time offset columns in the Security Investigator app.

This small, yet powerful feature allows engineers to speed up investigations and lock down event timestamps for easier navigation across their query results.

With reference time, you can efficiently navigate logs while maintaining an incident’s time context. It helps you track the time offset between events you’re analyzing and when the incident occurred. This allows you to uncover relevant threads and evidence, even from logs and events that might initially seem unrelated.

Reference time integrates diverse information, ensuring a consistent incident context across all data points.

For a more detailed example of how reference time speeds up threat hunting, please see the reference time use case in Dynatrace Documentation.

Get started

Visit Dynatrace Playground to see the reference time in action, or read more about how reference time speeds up threat hunting in Dynatrace Documentation.

___

© 2025 Dynatrace LLC

Dynatrace and the Dynatrace logo, are trademarks of the Dynatrace, Inc. group of companies. All other trademarks are the property of their respective owners.

The post Create context in Security Investigator with reference times appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/create-context-in-security-investigator-with-reference-times/feed/ 0
Duplicate cases: A game-changer for Security Investigator productivity and efficiency https://www.dynatrace.com/news/blog/duplicate-cases-a-game-changer-for-security-investigator-productivity-and-efficiency/ https://www.dynatrace.com/news/blog/duplicate-cases-a-game-changer-for-security-investigator-productivity-and-efficiency/#respond Fri, 07 Mar 2025 18:17:02 +0000 https://www.dynatrace.com/news/?p=68191 Dynatrace Security Investigator

Dynatrace introduces the next great addition to collaborative features for security investigators. You can now duplicate cases in Security Investigator. Why is duplicating a case document important? Imagine having a case from last month that you need to reuse again. You can now create an exact copy of the existing case in seconds, saving you […]

The post Duplicate cases: A game-changer for Security Investigator productivity and efficiency appeared first on Dynatrace news.

]]>
Dynatrace Security Investigator

Dynatrace introduces the next great addition to collaborative features for security investigators. You can now duplicate cases in Security Investigator.

Why is duplicating a case document important? Imagine having a case from last month that you need to reuse again. You can now create an exact copy of the existing case in seconds, saving you the hassle of starting from scratch. This promotes consistency, reduces errors, and streamlines your workflow.

Security Investigator dashboard in Dynatrace screenshot

Use cases for case duplication

Here are several additional use cases where duplicating cases can come in handy:

  • Duplicate an existing case for sharing: You’re in the middle of an investigation, and your colleague asks if they can assist you with a case. You can duplicate the case and share the duplicate with your coworker so you don’t lock each other out.
  • Create a snapshot of a current case: You’ve reached a critical point in an investigation, and to prevent accidental data loss, you want to create a quick snapshot of the investigation. So, you duplicate it and save a version of the case as a backup.
  • Duplicate a case shared with you: If a case is shared with you in read-only mode, locked by someone else, or you simply want to avoid meddling with another person’s case, you can duplicate it and continue working on it.

Duplicating cases isn’t just about making copies; it’s about empowering you to work smarter. With a single click, you can duplicate any case, customize it as needed, and focus on what matters most. It’s perfect for creating new cases or templates of existing cases easily and precisely.

With our recently announced Security Investigator templates, you can create templates from any shared case.

For example, a coworker shares a case with you that, following some minor adjustments can serve as a recurring investigation. You can duplicate such a case, edit it, and create a template for later use.

Get started

You can try out the new case-duplication functionality now on the Dynatrace Playground.

The post Duplicate cases: A game-changer for Security Investigator productivity and efficiency appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/duplicate-cases-a-game-changer-for-security-investigator-productivity-and-efficiency/feed/ 0
Reduce security incident response time with case templates https://www.dynatrace.com/news/blog/reduce-incident-response-time-with-case-templates/ https://www.dynatrace.com/news/blog/reduce-incident-response-time-with-case-templates/#respond Fri, 07 Mar 2025 18:04:16 +0000 https://www.dynatrace.com/news/?p=68182 VMware security advisory VMSA-2025-0004

We’re proud to announce that Dynatrace has introduced a new capability to speed up your security incident response and root cause analysis use cases with case templates in Security Investigator. Case templates allow you to start your investigations faster using prepared queries and evidence as a boilerplate. Repetitive tasks in security incident responses waste time […]

The post Reduce security incident response time with case templates appeared first on Dynatrace news.

]]>
VMware security advisory VMSA-2025-0004

We’re proud to announce that Dynatrace has introduced a new capability to speed up your security incident response and root cause analysis use cases with case templates in Security Investigator. Case templates allow you to start your investigations faster using prepared queries and evidence as a boilerplate.

Repetitive tasks in security incident responses waste time

When investigating incidents in production, engineers typically start each investigation with similar queries to understand what happened and where to look next, though the specifics can vary. Familiarity with the production environment’s artifacts (for example, environment names and deployment labels) is crucial, but gathering this information can be time-consuming, creating overhead before the actual investigation begins. The accumulated time spent on such activities before starting incident resolution can be a lot of overhead for engineers and the company.

Case templates assist in kicking off investigations

Dynatrace introduces case templates in Security Investigator to speed up new investigations and save engineers from manual, repetitive work.

Security Investigator Templates in Dynatrace screenshot

Case templates provide engineers with a boilerplate for their investigation. They offer ready-to-be-executed DQL queries and the required artifacts about your environment as evidence lists, saving time on manual query creation or copying the queries from incident response playbooks.

Craft templates in Security Investigator

Case templates can be created from existing investigations or downloaded from other sources, like Dynatrace blog posts or documented use cases. You can select any current case and create a template from it.

Security Investigator Case in Dynatrace screenshot

Once a template is created, you can adjust it via the Template Editor to better suit your needs.

Security Investigator Template editor in Dynatrace screenshot

Case templates can be downloaded from your environments for safekeeping and uploaded for use in different environments. They can be shared directly with other Dynatrace users or made available to everyone on the same tenant. You can download and upload templates from your tenant to facilitate wider adoption of templates in multi-tenant environments or share templates with others in the community.

You can learn more about case templates in Dynatrace documentation.

The post Reduce security incident response time with case templates appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/reduce-incident-response-time-with-case-templates/feed/ 0
Generate security events from Dynatrace Security Investigator via OpenPipeline https://www.dynatrace.com/news/blog/generate-security-events-with-security-investigator-via-openpipeline/ https://www.dynatrace.com/news/blog/generate-security-events-with-security-investigator-via-openpipeline/#respond Tue, 17 Sep 2024 16:31:41 +0000 https://www.dynatrace.com/news/?p=65615 Dynatrace Security Investigator via OpenPipeline

You’re in the middle of threat-hunting activities using Dynatrace and discover that some of your assets are trying to resolve a suspicious DNS name to extract data via covert channels like DNS tunneling. You now want to detect such events automatically by creating a custom Dynatrace security event. This blog post explains how to achieve […]

The post Generate security events from Dynatrace Security Investigator via OpenPipeline appeared first on Dynatrace news.

]]>
Dynatrace Security Investigator via OpenPipeline

You’re in the middle of threat-hunting activities using Dynatrace and discover that some of your assets are trying to resolve a suspicious DNS name to extract data via covert channels like DNS tunneling. You now want to detect such events automatically by creating a custom Dynatrace security event. This blog post explains how to achieve this goal.

Following the Route53 log use case in Dynatrace threat-hunting documentation, a DNS query record is structured in the following way:

{
    "version": "1.100000",
    "account_id": "684399999995",
    "region": "us-east-1",
    "vpc_id": "vpc-026e4d09999995",
    "query_timestamp": "2024-08-17T07:24:57Z",
    "query_name": "250607.56755a335668636d52634969497343694167496e4a6c59584e766269493649.434a4762334a696157526b5a5734694c416f6749434a6b5a5852686157787a.tiitha-maliciousdomain.com.",
    "query_type": "A",
    "query_class": "IN",
    "rcode": "NOERROR",
    "answers": [
        {
            "Rdata": "34.667.67.208",
            "Type": "A",
            "Class": "IN"
        }
    ],
    "srcaddr": "172.31.1.6",
    "srcport": "42241",
    "transport": "UDP",
    "srcids": {
        "instance": "i-0b56150a99995e6d",
        "resolver_endpoint": "rslvr-out-c8e7c49999999fb5a"
    }
}

Let’s say that the meaningful fields you need for the security event are:

  • timestamp when the DNS query was executed (query_timestamp);
  • source IP where the query was performed (srcaddr);
  • DNS query requested (query_name).

The JSON matcher, with its ability to parse out selected members, allows you to use the following simple DQL expression to extract only the fields you need:

json{
  timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp,
  string:query_name,
  ipaddr:srcaddr
}(flat=true)

As a result, you now have three additional fields with their respective types and values.

JSON matcher result with three additional fields with their respective types and values

To add more accuracy to the query you’re about to automate, you need to extract the domain portion from the query_name field. This can be achieved with the following DQL expression:

LD* '.'? ( (LD '.' LD):domain '.' EOS)

That DQL pattern translates to something like this:

“Please extract two line-data (LD) portions from the content separated by a dot LD '.' LD) that is immediately followed by a single dot ('.') and then reaches the end of the string (EOS). A single dot and some line data might precede it.”

So, the pattern will match the highlighted DNS names below:

highlighted DNS names

Adding a filter for the correct domain and limiting the query to only the last 5 minutes gives you the following DQL query to automate:

fetch logs, from: -5m
| filter aws.log_group == "/aws/route53/unguard-secla-demo/resolver-logs"
| parse content, "json{timestamp('yyyy-MM-DDTHH:mm:ssZ'):query_timestamp, string:query_name, ipaddr:srcaddr}(flat=true)"
| parse query_name, "ld* '.'? ( (ld '.' ld):domain '.' eos) "
| filter domain == "tiitha-maliciousdomain.com"
| fields query_timestamp, srcaddr, query_name

Automate the query

Dynatrace AutomationEngine provides the ability to automate such periodic query executions. When you’re satisfied with your DQL query and are ready to automate it, select the Open with button in the lower right-hand corner of the query window.

Dynatrace AutomationEngine DQL query

As a result, a modal dialog will display, providing you with different possibilities about where to open the DQL query. In the current use case, you should open the query in Workflows.

Workflows in Dynatrace

Modify the default workflow

To execute the workflow automatically every 5 minutes, change the default on-demand trigger to either a cron or time-interval trigger. You can easily achieve this by selecting the trigger box in Workflows and following the instructions in the right-hand pane. You can read more about workflow triggers in Workflow schedule trigger documentation

Workflow schedule trigger

Ingest query results as security events

The simplest way to do this is to use Dynatrace OpenPipeline. OpenPipeline allows you to create custom endpoints for data ingestion and process the events in the pipeline (for example, adding custom pipe-dependent fields to simplify data analysis in a later phase).

Set up a custom pipeline

The best way to set up a security event ingestion to Dynatrace is via Dynatrace OpenPipeline. You can create your own ingest endpoint that will then perform custom processing activities, create additional fields, store these events in separate buckets, and much more. For the current use case, we can create a custom security event pipeline to which we can send all the automated security events we have created from the Security Investigator, add respective fields for future analysis, and store them in the default security events bucket.

Now, create a general ingest endpoint for this use case and any automated detections we want to initiate from the Security Investigator in the future.

  1. Go to OpenPipeline.
  2. Choose Events > Security Events.
  3. On the Ingest Sources tab, select + Source.
  4. Give your pipeline a name, for example, Automated discovery from Security Investigator.
  5. Choose a path to the new ingest source, for example, Signals.
  6. Since we currently don’t want to create any dynamic rules to our pipeline, leave Routing set to static and choose the Custom events pipeline as the route.Ingest Sources in Dynatrace screenshot
  7. Go to the Pre-processing tab and add a new processor with the type Add Fields.
  8. Give the processor a name, for example, My custom processor.
  9. Add the fields that you want to be automatically added when an event passes the pipeline. In our case, we will add the following fields:
    – kind == “SECURITY_EVENT”
    – provider == “Automated Detection Script”
    – category == “AUTOMATED_DETECTION”
    – provider_product == “Security Investigator App”
  10. To figure out which fields to add for custom security events, refer to the semantic dictionary.
    Following the overall convention here will simplify analyzing this data later, together with other security events created by other sources.Custom processor in Dynatrace screenshot
  11. You can test the processor with sample data to see what kind of event will be created based on the processor, and if you’re happy with the result, select Save.

For the current demo, this is enough, however, there are lots of things that you could configure additionally. Take a look at the Ingest custom security events via API documentation to learn more about sending security events to OpenPipeline.

Create and secure the token for event ingest

To create the token needed to ingest events, follow the steps in How to ingest data (events) documentation. When you have created the token, we recommend that you protect it in the Credential Vault instead of just using it in plain text in the automation engine.

  1. Open Credential Vault.
  2. Select Add new credential.
  3. Select token as the credential type and give it a meaningful name.
  4. Select App Engine as the credential scope.
  5. Paste your ingest token into the token field and select Save.

After saving the credentials, an ID is presented to you in the form of CREDENTIALS_VAULT-12345. Write this down for later usage.

Ingest records through the OpenPipeline custom endpoint

Let’s return to our workflow and add a follow-up task for the DQL query. Choose the JavaScript task type and start writing the code to ingest security events.

Consider the following code for your function:

import { execution } from '@dynatrace-sdk/automation-utils';
import { credentialVaultClient } from '@dynatrace-sdk/client-classic-environment-v2';

export default async function ({ execution_id }) {

  const ex = await execution(execution_id);

  // load the token from the credential vault by its' ID
  const token = await credentialVaultClient.getCredentialsDetails({
    id: "CREDENTIALS_VAULT-F12345",
  }).then((credentials)=> credentials.token);

  // loop through the DQL query results and ingest them one by one
  var result = await ex.result('execute_dql_query_1');
  result.records.forEach (record => {
    ingest_event(token, record);
  });
}

async function ingest_event(token, payload) {

  payload["event.description"] = "Suspicious DNS query from AWS DNS Logs";

  fetch('https://<dynatrace_env>.dynatrace.com/platform/ingest/custom/events.security/signals', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json; charset=utf-8",
      "Authorization": `Api-Token ${token}`
    },   
    body: JSON.stringify(payload)
  }).then(function(response) {
    console.log(response);
  })
}

The code loads the token from the Credential Vault that will be used to ingest the events to OpenPipeline. Then, we iterate through the result set we got from the execute_dql_query_1 task and pass each event along with the ingest token to the ingest function, which will POST the event to the OpenPipeline ingest endpoint that we created earlier. The response to the request is shown in the console for debugging purposes.

We’re also adding a custom field called event.description to the event. You can add custom fields to your event from the OpenPipeline processing as well, but since we want to keep the ingest source as general as possible, it makes more sense to add the event-specific fields near the event, not the pipeline.

Now, give it a distinct name and test if it runs perfectly. Ingesting events to Grail via OpenPipeline should result in a status: 202 that is visible in the run_javascript_1 log console.

Detect malicious DNS queries

Fetch the security events from Grail

To view the newly created security events, follow the same principles as with fetching any other security data from Grail. Using the following DQL query, you can see all the events you have started to ingest:

fetch events
| filter event.kind == "SECURITY_EVENT"
| filter event.category == "AUTOMATED_DETECTION"

To feel the power of the semantic dictionary and to illustrate what you gain from using it, consider the following query. If you want to visualize all the security events ingested to Grail over the last 12 hours by event.category, it is quite simple with the following query thanks to the unified field names:

fetch events, from: -12h
| filter event.kind == "SECURITY_EVENT"
| makeTimeseries count(), by: { event.category }, interval:1h

The result could be a line chart like this:

Security events line chart

Of course, you can follow your own naming convention, but this would make it much harder to view all the security events (both custom and Dynatrace-created events) in conjunction.

Operationalize the results

You can create additional workflows for your security events in Grail. For example, you can create a workflow that notifies you in Slack when a new event is created or integrate your workflows with Jira to create a ticket every time such events arise.

You can create a custom integration to your own SIEM system (similarly as we did for the OpenPipeline) for events that can enrich your security data in other environments. Consider your use case and work processes to determine the best way to approach security events available in Grail.

Ready to give this a shot yourself? Run Security Investigator and try out the same workflow. Then, post your feedback or questions in the Dynatrace Community.

The post Generate security events from Dynatrace Security Investigator via OpenPipeline appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/generate-security-events-with-security-investigator-via-openpipeline/feed/ 0
Collaborate with peers in hunting security threats https://www.dynatrace.com/news/blog/collaborate-with-peers-in-hunting-security-threats/ https://www.dynatrace.com/news/blog/collaborate-with-peers-in-hunting-security-threats/#respond Thu, 01 Aug 2024 15:34:09 +0000 https://www.dynatrace.com/news/?p=65081 Observability graphic

In the dynamic world of security investigations, efficient collaboration can make all the difference. Today, we're thrilled to announce that Dynatrace has enabled case sharing in the Security Investigator app, which will transform how professionals conduct collaborative investigations.

The post Collaborate with peers in hunting security threats appeared first on Dynatrace news.

]]>
Observability graphic

When kicking off a threat hunting activity, you can immediately share your investigation with your teammates, to keep them up-to-date and allow them to collaborate with the ongoing investigation.

Sharing a case

There are several ways to share a case:

  • Personal or group sharing – Select colleagues by name or a group to share the case.
  • Link sharing – Generate and distribute a shared link in Slack or include it in your report.

Both share modes support either read-only or edit privileges. Read-only mode allows you to browse the case. Edit mode allows you to execute queries and modify the case and its contents.

You can also combine the modes: You can create a read-only link to distribute the case in the organization for everyone to view and give edit privileges to your teammates for collaborative investigations at the same time!

Handle shared cases

You can identify shared cases by a small Shared icon on the main page of Security Investigator. Cases you have shared are marked with a blue icon; cases shared with you include a white icon. Hover over the Shared icon for more details about who shared the case with you. If the case is shared with you in read-only mode, you’ll see the respective label next to the icon.

Empower your investigations with joint editing

When you grant your teammates edit access, they can perform any investigative action, from executing queries and creating new branches to modifying the query tree or removing evidence from the case.

To avoid integrity issues, only one editor is allowed at a time. When an investigator opens a case for editing, it’s read-only for all other investigators, even if they have edit privileges or are the case owners; other investigators see a notification stating that the case has been locked and who is currently dealing with it.

Control access permissions

When you grant your teammates edit access, they can perform investigative actions within the case, but the ownership remains the same. Only the owner can grant and revoke sharing permissions or delete the case.

If the shared link has been distributed too widely or leaked, or the owner wants to revoke link access to the case, you can always remove or recreate it. The old link will no longer provide access to the case.

What’s next

For more details, check out Security Investigator in Dynatrace Hub.

The post Collaborate with peers in hunting security threats appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/collaborate-with-peers-in-hunting-security-threats/feed/ 0
RegreSSHion vulnerability: Detecting CVE-2024-6387 in OpenSSH https://www.dynatrace.com/news/blog/detecting-regresshion-with-dynatrace/ https://www.dynatrace.com/news/blog/detecting-regresshion-with-dynatrace/#respond Tue, 02 Jul 2024 16:00:01 +0000 https://www.dynatrace.com/news/?p=64548 Apache Struts CVE-2024-53677; regreSSHion vulnerability

What is CVE-2024-6387 (regreSSHion vulnerability)? The regreSSHion vulnerability (CVE-2024-6387) has been discovered by Qualys Threat Research Unit (TRU) as a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH server (sshd) in glibc-based Linux systems. The vulnerability, a signal handler race condition in OpenSSH server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based […]

The post RegreSSHion vulnerability: Detecting CVE-2024-6387 in OpenSSH appeared first on Dynatrace news.

]]>
Apache Struts CVE-2024-53677; regreSSHion vulnerability

What is CVE-2024-6387 (regreSSHion vulnerability)?

The regreSSHion vulnerability (CVE-2024-6387) has been discovered by Qualys Threat Research Unit (TRU) as a Remote Unauthenticated Code Execution (RCE) vulnerability in OpenSSH server (sshd) in glibc-based Linux systems. The vulnerability, a signal handler race condition in OpenSSH server (sshd), allows unauthenticated remote code execution (RCE) as root on glibc-based Linux systems, which is a significant security risk. This race condition affects sshd in its default configuration.

Technical Overview

  • Name: CVE-2024-6387
  • Description: A security regression (CVE-2006-5051) was discovered in OpenSSH’s server (sshd). There is a race condition which can lead sshd to handle some signals in an unsafe manner. An unauthenticated, remote attacker may be able to trigger it by failing to authenticate within a set time period.
  • CVSS Base Score: 8.1 HIGH
  • Source: CVE (at NVD; CERT, LWN, oss-sec, fulldisc, Red Hat, Ubuntu, Gentoo, SUSE bugzilla/CVE, GitHub advisories/code/issues, web search, more

What is OpenSSH?

OpenSSH is an essential tool for secure communication and administration in Unix-like operating systems and is also available for Windows. It is part of the broader OpenBSD project and is widely adopted due to its robustness and security features.

What is the potential impact of regreSSHion vulnerability?

Exploiting this vulnerability can lead to full system compromise, where an attacker can execute any commands or code with the highest privileges. This can result in a complete system takeover, malware installation, data manipulation, and the creation of backdoors for persistent access. Moreover, gaining root access will enable attackers to bypass critical security mechanisms such as firewalls, intrusion detection systems, and logging mechanisms, further obscuring their activities. This can result in significant data breaches and leakage, giving attackers access to all data stored on the system, including sensitive or proprietary information that can be stolen or publicly disclosed.

To avoid sounding too alarming, we have to emphasize that this vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. As written in the Qualys white paper, “In our experiments, it takes ~10,000 tries on average to win this race condition. ” To our knowledge as of writing this article, no properly working exploits have been created.

How to mitigate the risk of CVE-2024-6387

The following versions of OpenSSH are impacted by this vulnerability:

  • Versions earlier than 4.4p1 (unless patched for CVE-2006-5051 and CVE-2008-4109)
  • Versions from 8.5p1 up to, but not including, 9.8p1

OpenBSD systems are unaffected by this, as OpenBSD developed a secure mechanism in 2001 that prevents this vulnerability.

There are three strategic paths to take to mitigate this threat:

  • Patch Management: Quickly apply available patches for OpenSSH or upgrade to the relevant version that is not impacted by this vulnerability
  • Access Control: Limit SSH access via network-based restrictions
  • Network Segmentation: Divide networks to restrict unauthorized access and lateral movements from other systems

How to detect regreSSHion exploitation attempts with Dynatrace

This vulnerability is challenging to exploit due to its race-condition nature, requiring multiple attempts for a successful attack. Thanks to this noisy approach it can be detected from your sshd logs with Dynatrace DQL queries using Dynatrace Security Investigator.

Look for timeout events

Exploitation attempts for this vulnerability can be identified by many lines of “Timeout before authentication” in the logs. The log line in the logs could look something like this:

Jul  1 09:40:39 [localhost] sshd[787404]: fatal: Timeout before authentication for 192.168.12.45 port 17296

To search for such events from your sshd authentication logs stored in Grail, the following DQL query can be used:

fetch logs, from: -48h
| filter contains(content, "Timeout before authentication")

If you don’t get any results, that might be good news! You can further expand the timeframe of your query by changing the from time to -7d. This will get you logs from last week to make sure that no one tried to exploit the system during the last week.

Look for a high volume of log entries

There’s a chance that someone has tried to attack your system with a different attack or that the attack did not succeed as intended. For example, some Proof-of-concept attacks have failed, and these failures write various error messages to the victims’ sshd logs. It’s always good to check the log trend line in general to detect such attacks and when hunting for the unknown.

It’s easy to create a timeseries from your logs using DQL. With the following query, you can create a chart that shows sshd log line count from the previous day, grouped by host and aggregated at an hourly interval.

fetch logs, from: -24h
| filter contains(content, "sshd[")
| makeTimeseries count(), by: { dt.entity.host }, interval:1h

From the chart, it’s easy to see that nothing anomalous was detected using this query: no weird peaks or anomalies could be extracted from the chart.

Chart that shows sshd log line count from the previous day, grouped by host and aggregated at an hourly interval in Dynatrace

Look for suspicious IP addresses in sshd logs

To ensure no one has tried to access your services from a suspicious location, it’s valuable to extract all the IP addresses from your sshd logs and see if something suspicious emerges from there. To do that, the following DQL query is helpful:

fetch logs, from: -2d
| filter contains(content, "sshd[")
| parse content, "ARRAY{ ld* ipaddr:ip }{0,10}:source_ip"
| expand source_ip
| summarize count(), by:source_ip
| sort source_ip desc

This query will fetch your sshd logs from the past two days, will extract ALL the IP addresses (both ipv4 and ipv6) from the content and will sort the result based on their occurrances to your results. If you discover any suspicious IP addresses from this list, you can start your investigation based on the discovered threat! If you see only your own addresses, you can rest assured: your sshd has not been accessed by an unknown IP address.

Analyze network flow logs

Last but not least, your network logs are the ultimate source of data. If you’re using AWS for your cloud platform and you have forwarded your VPC flow logs to Dynatrace, you can use DQL to look for your potential victims AND suspicious IP addresses that have tried to access your servers using SSH.

Consider the following query:

fetch logs, from: -24h
| filter aws.log_group == "/aws/vpc/flow_logs"
| parse content, """INT:version ' '
NSPACE:account_id ' '
NSPACE:interface_id ' '
('-' | IPADDR:src_addr) ' '
('-' | IPADDR:dst_addr) ' '
('-' | INT:src_port) ' '
('-' | INT:dst_port) ' '
('-' | INT:protocol) ' '
('-' | LONG:packets) ' '
('-' | LONG:bytes) ' '
TIMESTAMP('s'):start ' '
TIMESTAMP('s'):end ' '
LD:action ' '
LD:log_status"""
| filter dst_port == 22

The query will fetch all the Grail logs ingested from the CloudWatch log group called /aws/vpc/flow_logs. Using the VPC flow log default pattern available in DPL Architect, we can extract the meaningful fields to see only the network traffic targeting the SSH port.

From this dataset, we can create metrics based on the destination IP address to detect all the potential victims by adding the following code snippet to the previous query:

| filter action == "ACCEPT"
| makeTimeseries count(), by: {dst_addr}, interval: 1min

Viewing the results as a graph, you can see that some servers have received an anomalous amount of SSH traffic towards them! From here, you can continue to look further into the specific destinations to understand if this is malicious activity.

Chart showing servers have received an anomalous amount of SSH traffic towards them in Dynatrace

To understand who is targeting you, a similar query can be used. Instead of dst_addr use a src_addr to see the most active sources that have been targeting any of your ssh daemons on all the servers in the respective VPC.

| filter action == "ACCEPT"
| makeTimeseries count(), by: {src_addr}, interval: 1min

The results show that of the 725 SSH sources, three stand out with a huge load of requests that require further analysis.

Chart that shows that of the 725 SSH sources, three stand out with a huge load of requests that require further analysis in Dynatrace

Next steps for mitigating regreSSHion vulnerability

This particular vulnerability is challenging to exploit due to its race condition nature. However, all vulnerabilities should be taken seriously, their potential impact on your environment analyzed, and, where possible, mitigated. If you plan not to take any action for whatever reason, you should at least analyze the potential risk and make an informed decision before taking any action.

Get article updates or report security vulnerabilities

Dynatrace takes a proactive approach in communicating security vulnerability information to customers. Learn more about Dynatrace security and our security policy. To report a security issue, email security@dynatrace.com.

The post RegreSSHion vulnerability: Detecting CVE-2024-6387 in OpenSSH appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/detecting-regresshion-with-dynatrace/feed/ 0
Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator https://www.dynatrace.com/news/blog/speed-up-evidence-driven-security-investigations-and-threat-hunting-with-dynatrace-security-investigator/ https://www.dynatrace.com/news/blog/speed-up-evidence-driven-security-investigations-and-threat-hunting-with-dynatrace-security-investigator/#respond Thu, 01 Feb 2024 14:00:32 +0000 https://www.dynatrace.com/news/?p=61834 Query Tree Filter

Dynatrace Security Investigator is a new application on the Dynatrace platform dedicated to security operations and security analysts. The evidence-driven approach to the data in Dynatrace Grail™ enables security teams to achieve faster investigations and obtain accurate results without losing the data context of investigations. The Security Investigator app is designed for evidence-driven use cases such as threat hunting and incident solving.

The post Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator appeared first on Dynatrace news.

]]>
Query Tree Filter

As a security analyst investigating security incidents or threat hunting, you often must navigate among multiple executed queries and their respective results (in other words, investigation steps). You also need to manage the evidence you gather during investigations and reuse it when building additional queries. All this must be as fast and seamless while maintaining the investigation context.

Navigating investigations and threat hunting can involve saving executed queries and found evidence in unstructured text documents that can quickly lose their context. Manually documenting investigation steps can lead to human errors, like forgotten or wrongly documented queries or evidence, which will result in missed threats and wasted time. As the investigation grows and the different queries accumulate, it becomes harder to maintain the throughlines among the different branches.

The Dynatrace Security Investigator app provides branching navigation, making it easy to track your path through investigations.

Investigations in Grail are conducted using the Dynatrace Query Language (DQL). For multiple investigation steps, the Security Investigator provides a representation of the steps in a tree-like view. With this visualization, you can keep track of your investigation and navigate to previous steps and respective results in the investigation history—all in context, with no need to re-execute anything.

Security Investigator builds a query tree for every stage in your investigation.
Figure 1. Security Investigator builds a query tree of all the stages in your investigation.

Every new DQL query execution creates a new descending node while keeping the integrity and hierarchy of previous queries and results unchanged.

Easily track threat-hunting twists and turns

Threat hunting is a nonlinear process. When looking for the unknown, you often stumble upon an unexpected thread of evidence that sparks a new idea and takes the investigation in a new direction. To support this nonlinear approach, we’ve created the investigation branching feature. When modifying and re-executing a previous step, the app creates a new branch while keeping the original investigation path intact and visible in the query tree.

You can create any new branches for new ideas and hypotheses while retaining the ability to navigate back to your initial train of thought. With the ability to customize and name branches, you can simplify locating historical steps without having to click through them individually.

The final visual representation provides a comprehensive overview of the entire investigation. You can see in a structured way how you reached your conclusions without needing to manually document the investigation flow.

With Security Investigator, you can always find the last working query, which was easy to lose in the past.

Character precision on a petabyte scale

Security Investigator increases the speed of investigation flows and the precision of evidence, leading to higher efficiency and faster results. Both outcomes are game-changers in time-critical situations, where answers are expected “yesterday.”

View raw content details when you need them

Data ingested into Grail is kept in its original format. A schema is applied only during query execution, or as we call it, “schema-on-read.” This makes data immune to format changes and enables you to choose your data structure based on your use cases.

Drill down on content details to view data in its original format.
Figure 2. Drill down on content details to view data in its original format.

The Security Investigator app enables you to view all data in its original format, regardless of the content, including characters that other tools might hide or not interpret. Enabling the multi-line content details view shows the stack traces with their line breaks in their original form, enabling you to understand the data much faster.

Multi-line content shows stack traces with line breaks in their original form.
Figure 3. Multi-line content shows stack traces with line breaks in their original form.

Find the needle in the “needle stack”

Finding a small metallic object in a stack of non-magnetic objects is relatively easy. Finding a particular object among millions of similar objects is trickier. With Security Investigator’s flexible filtering, you can create accurate DQL query filters quickly and efficiently.

Select a portion of a text string to add to your DQL query with the correct syntax for the desired action.
Figure 4. Select a portion of a text string to add to your DQL query with the correct syntax for the desired action.

If you need to fetch raw content from Grail and filter the results based on a selection of the content, you don’t need to copy partial records and manually paste them into an appropriate filter. Instead, you can easily select the portion of the text string you need and add it to your DQL query directly from the context menu.

To add multiple values to a filter, you can either select individual values or a range of values and apply them as a single filter in your DQL query. Only the unique values from the selection are added to the query, so you don’t have to investigate unneeded values.

You can select multiple values to add to your DQL query without having to investigate unneeded values.
Figure 5. You can select multiple values to add to your DQL query without having to investigate unneeded values.

Use your findings to enrich your investigation

With Security Investigator evidence management, you can persist relevant findings by attaching them to your investigation. Whether it is a suspicious IP address or a potential indicator of compromise, you can attach it to the investigation and use it later.

You can create type-specific filters to enable fast and accurate results, either focusing only on suspicious activities or filtering out the safe events that don’t require your attention.

Continue hunting for the unknown

Security use cases such as threat hunting and incident analysis are nonlinear, evidence-driven activities in which you don’t know what you’re looking for. You’re not pursuing a specific target; rather, you’re looking for the unknown. With Security Investigator, Dynatrace enables you to be more efficient, investigate faster, and be more precise.

Use Dynatrace Security Investigator for faster and more precise evidence-based security investigations.

Use filtering to narrow down results and focus your research.
Figure 6. Use filtering to narrow down results and focus your research.

The post Speed up evidence-driven security investigations and threat hunting with Dynatrace Security Investigator appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/speed-up-evidence-driven-security-investigations-and-threat-hunting-with-dynatrace-security-investigator/feed/ 0
Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL https://www.dynatrace.com/news/blog/detect-vmware-aria-operations-for-logs/ https://www.dynatrace.com/news/blog/detect-vmware-aria-operations-for-logs/#respond Wed, 25 Oct 2023 12:00:43 +0000 https://www.dynatrace.com/news/?p=60261 New SQL injection vulnerability in FileCatalyst Workflow

Earlier this week, virtualization services provider VMware alerted customers to the existence of a proof-of-concept (PoC) exploit for a recently patched security flaw in VMware Aria Operations for Logs.

The post Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL appeared first on Dynatrace news.

]]>
New SQL injection vulnerability in FileCatalyst Workflow

VMware Aria Operations for Logs (formerly known as vRealize Log Insight) is used across enterprises to collect logs and provide analytics. The company recently announced a high-severity vulnerability in an earlier version of the tool. Tracked as CVE-2023-34051 (CVSS score: 8.1), the Aria Operations for Logs vulnerability relates to a case of authentication bypass that could lead to remote code execution. Basically, this means that an unauthenticated malicious actor could inject files into the operating system of an impacted appliance, which can result in remote code execution.

James Horseman from http://Horizon3.ai and the Randori Attack Team have been credited for the discovery and reporting. They have made a PoC for the vulnerability available and published the relevant indicators of compromise (IoC).

This vulnerability is a so-called “patch bypass” for a set of critical flaws that were addressed by VMware earlier this year, also discovered by Horizon3.ai. In their report, they presented how an attacker could use three different CVEs to achieve remote code execution. Since the patch only blocks access to Thrift services by IP and does not fix the other CVEs in VMSA-2023-0001, all an attacker needs to do is spoof an IP address and use the published attack again.

In this blog post, we show how to discover the original attacks toward the Aria Operations for Logs vulnerability using Dynatrace and DQL by finding the IoC-s from the log records.

How to detect VMware Aria Operations Logs exploitation with Dynatrace

Since we’re talking about a patch bypass, we’re looking at the original attack vector described in this blog post. To exploit this vulnerability to gain remote code execution (RCE), the following steps have to be taken by the malicious actor:

  1. Create a Thrift client to gain unauthenticated access to the Log Insight Thrift server.
  2. Spoof the IP address of the known worker.
  3. Create a malicious TAR file containing a directory traversal using a valid PAK file.
  4. Using remotePakDownloadCommand, upload the malicious PAK file to /tmp/<filename>.pak.
  5. Extract the file using pakUpgradeCommand. This writes the file to where it’s needed on the filesystem.

A technical deep dive about the attack can be found in this article, VMware vRealize Log Insight VMSA-2023-0001 Technical Deep Dive.

Discovering the attack with Dynatrace

To discover this attack, you have to go through the log files that Aria Operations for Logs creates. The Log Insight server stores relevant logs in the file /var/log/loginsight/runtime.log. This log file is used to track all runtime information related to Log Insight. To analyze these logs with Dynatrace and DQL, the logs need to be ingested into Dynatrace beforehand.

Forwarding logs away from the main system is beneficial in case of any kind of successful attack by a malicious actor: if an attacker decides to remove all evidence from the system (including log files), then a security investigator wouldn’t have traces left to analyze. Sending logs to Dynatrace at runtime safeguards the log files for later analysis in a remote and secure location.

Security-related log records of Aria Operations for Logs are structured in the following format:

[2023-10-25 11:28:29.709+0000] ["https-jsse-nio-443-exec-9"/10.153.234.136 DEBUG] [com.vmware.loginsight.web.actions.misc.LoginActionBean] [User login success: vIDM: SAM=myusername, Domain=vmware.com, UPN=myusername@vmware.com]

When a new remote PAK file is downloaded as a tarball, the following log lines are created:

[com.vmware.loginsight.daemon.commands.SystemCommands] [PAK download initiated by node f2449ed5-11ee-45fd-a0a0-6225a33a8ac6] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Deleting existing PAK file: /tmp/exploit.pak] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak] 
[com.vmware.loginstght.daemon.commands.SystemCommands] [Max allowed pack size is 1505916450 bytes] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Current downloaded size is 21910]
[com.vmware.loginsight.daemon.commands.SystemCommands] [Total downloaded size is 21910]

From these log lines, the best candidate to look for in the logs is the third populated log, Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak].

Now, let’s fire up the DPL Architect and create a suitable pattern for this line:

MVware Aria Operations for Logs DPL example

With the created DPL pattern, you can extract the timestamp field ts, the logger class name as class that populated the log lines and the action that was written to the log record. These extracted fields give you the option to filter out the relevant records:

MVware Aria Operations for Logs DPL example to filter out records

However, there’s another goal to keep in mind: you need all the log records that contain the Downloading statement and URL from where the tarball was loaded and the destination path in /tmp/.

As the blog post states: “The URL and filename will likely be different. However, the filename will always have the format /tmp/<filename>.pak. Also, note that this log may be legitimate. To determine if an attack has occurred, you need to have your administrator evaluate the URL to determine if it is a legitimate download URL.”

Let’s continue with our DPL pattern and extract the full URL where the payload was downloaded and the relevant filenames. Again, we’re opening up the DPL Architect and creating the following pattern:

'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file

With the created pattern, we’re extracting the url and the tarball’s name as download_file into separate fields using the look-around functionality of DPL. Additionally, we’re extracting the filename from the destination file path as well as the whole saved_file value for simpler analysis.

DPL pattern extracts the following result set from the log record:

DPL Architect screenshot

Applying this pattern and filtering the records as described in the blog post gives you the following DQL query:

fetch logs
| filter log.source == "/var/log/loginsight/runtime.log"
| parse content, """'[' timestamp('yyyy-MM-dd HH:mm:ss.SZ'):ts '] [' ld '] [' ld:class '] [' ld:action ']'"""
| filter class == "com.vmware.loginsight.daemon.commands.SystemCommands"
| parse action, """'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file"""
| filter download_file == filename
| fields ts, url, saved_file

The query produces the following result, which you can now pass on to your administrator for manual evaluation.

DPL query result screenshot

A longer description of how the VMware Aria Operations for Logs exploitation works can be found in the Horizon3 blog post.

Further reading about how to use DPL Architect for security use-cases can be found at Speed up your security investigations with DPL Architect.

How to mitigate the Aria Operations for Logs vulnerability exploit

To mitigate this exploit, update to the latest version of VMware Aria Operations for Logs by applying the latest patches. Or follow the instructions within the VMSA-2023-0021.

Sources

  1. VMware vRealize Log Insight VMSA-2023-0001 IOCs
  2. GitHub – horizon3ai/CVE-2023-34051: VMware Aria Operations for Logs CVE-2023-34051
  3. VMware Aria Operations for Logs CVE-2023-34051 Technical Deep Dive and IOCs
  4. VMSA-2023-0021
  5. Alert: PoC Exploits Released for Citrix and VMware Vulnerabilities

The post Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/detect-vmware-aria-operations-for-logs/feed/ 0
Speed up your security investigations with DPL Architect https://www.dynatrace.com/news/blog/speed-up-your-security-investigations-with-dpl-architect/ https://www.dynatrace.com/news/blog/speed-up-your-security-investigations-with-dpl-architect/#respond Thu, 12 Oct 2023 18:33:12 +0000 https://www.dynatrace.com/news/?p=60004 DPL Architect

Grail™, the Dynatrace causational data lakehouse, offers you instant access to any kind of data, enabling anyone to get answers within seconds. As all historical data is immediately available in Grail, no data rehydration is needed, even for analysis of suspicious timeframes that are lengthy or ancient.

The post Speed up your security investigations with DPL Architect appeared first on Dynatrace news.

]]>
DPL Architect

To help you raise the quality of your investigation results, Dynatrace offers an easy way of structuring data using DPL Architect. This tool lets you quickly extract typed fields from unstructured text (such as log entries) using the Dynatrace Pattern Language (DPL), enabling you to extract timestamps, determine status codes, identify IP addresses, or work with real JSON objects. This allows you to answer even the most complex questions with ultimate precision. The best thing: the whole process is performed on read when the query is executed, which means you have full flexibility and don’t need to define a structure when ingesting data.

>> Scroll down to see Dynatrace DPL Architect in action (24-second video)

Investigating log data with the help of DQL

Let’s look at a practical example. The simplest log analysis use cases can be solved in seconds by applying a simple filter to the query results. If a CISO asks, “Have we seen the IP address 40.30.20.1 in our logs within the last year?” a simple DQL query seems to suffice:

fetch logs, from: -365d
| filter contains(content, “40.30.20.1”)

This query returns all the records that contain the string value 40.30.20.1.

In the next step, adding DQL aggregation functions enables you to answer more complex questions like “How many times in an hour has this IP address visited our website within the last two weeks?” Again, a simple DQL query helps you out:

fetch logs, from: -14d
| filter contains(content, “40.30.20.1”)
| summarize by: bin(timestamp, 1h), count()

However, this kind of simple filtering using a sub-string search is prone to errors and is not precise enough. The mentioned filters return all records that contain 40.30.20.1, but are not necessarily from the source IP portion of the log format. The string 40.30.20.1 might also appear elsewhere, for example, in query parameters, and a simple string filter will return all records containing this search term:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395
40.30.20.109 - - [28/Aug/2023:10:22:04 -0300] "GET / HTTP/1.1" 200 2216

The first log record is matched because of its query parameter—something we don’t care about in our case. The second log record came up because the source IP contains the IP address—this is what we’re really looking for. In a nutshell, if you’re querying terabytes of logs and wish to drill down to specific records, filtering strings from plaintext log content is not enough.

The issue is that questions from CISOs aren’t usually so trivial when it comes to security use cases. Or, the log format where the answers should be looked for is more complex (for example, AWS CloudTrail logs). As a consequence, we need to search structured data to avoid mistakes. To minimize false positives, increase the precision of queries, and get the maximum out of DQL, fields can be extracted from record content.

Extracting patterns using DPL

Dynatrace Pattern Language (DPL) is a pattern language that allows you to describe a schema using matchers, where a matcher is a mini-pattern that matches a certain type of data. These matchers can be used to extract new fields from the schemaless data in Grail to enable more precise log filtering. Consider the same Apache access log example from above:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395

To extract the client IP addresses from the beginning of the log, a simple DPL pattern can be used:

IPADDR:client_ip

The IPADDR matches both versions of IP addresses (IPv4 addresses in dot-decimal notation and IPv6 addresses in hextet notation), leaving all other IP-like strings unmatched. For example, if the log record contains a dot-decimal number that is NOT an IPv4 address (999.999.999.999), it won’t be matched.

DPL patterns can be applied in DQL using the parse command. The simplest way to get started with DPL is to use Dynatrace DPL Architect.

DPL Architect to the rescue

DPL Architect is a handy tool, accessible through the Notebooks app, which supports you in quickly extracting fields from records. It helps create patterns, provides instant feedback, and allows you to save and reuse DPL patterns, for faster access to data analytics use cases.

To open the DQL Architect, you have to execute a DQL query, select the content, and choose Extract fields.

Video thumbnail

Figure 1: Extract fields in Notebooks using DPL Architect (24-second video)

Starting with preset patterns

The simplest way to extract data is using one of the ready-to-use preset patterns available for the most popular technologies, such as AWS, Microsoft, or GCP. Start exploring AWS VPC Flow logs or analyzing Kubernetes audit logs by choosing the patterns from the panel on the left side. You can also customize the list by adding your own individual patterns.

Figure 2: Choose from a list of available patterns
Figure 2: Choose from a list of available patterns

Developing a new pattern using DPL Architect

DPL Architect helps you by creating your own patterns and provides instant feedback. Start typing a DPL expression in the pattern field, and review the matching data (highlighted below) in the preview editor.

Figure 3: Match preview editor
Figure 3: Match preview editor

For accessing the extracted fields in the resultset, it’s necessary to add a name. In the example below, we’re only interested in the client_ip and the related response_code. The pattern matches the whole record, but only two fields are being extracted since they have defined extract names. Any other data between those two fields won’t be visible in the results (however, it is very easy to add them later if necessary).

Figure 4: Match preview editor
Figure 4: Match preview editor
Figure 5: Review extracted fields in Results tab
Figure 5: Review extracted fields in the Results tab

Remember, we started our journey with the DPL Architect by selecting query results in a notebook. The colored bar at the top of the DPL Architect shows how many records from your query result match the current DPL pattern. This enables you to modify the DPL pattern to ensure it matches all records in the resultset.

Figure 6: Add records to the Match preview that don't match the current pattern.
Figure 6: Add records to the Match preview that don’t match the current pattern.

Records that don’t match the current pattern can easily be added to the Match preview panel by selecting Add to preview.

Figure 7: Add unmatching records to Match preview dataset
Figure 7: Add unmatching records to the Match preview dataset

After changing the DPL pattern, any previously unmatched records will be highlighted and the progress bar on the top informs you that 100% of the records in the Notebooks resultset are matched. It’s now time to insert our pattern into the DQL query by selecting Insert Pattern.

Figure 8: Review the result and insert the created pattern
Figure 8: Review the result and insert the created pattern

Precise extraction of fields from complex data

DPL also provides matchers for more complex data structures, like key-value pairs, structures, and JSON objects, that enable you to parse individual sub-elements from the whole object. Consider the following log record:

1693230219 230.4.130.168 C "GET / HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" 186 {"username":"james","result":0 }

Imagine your CISO comes with a request: “Give me the list of all IP addresses where the user ‘james’ has successfully logged in from.” Extracting these three fields and building a DQL query is as easy as pie:

LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)

The DPL will extract three fields: an IP address from the record, a string from the JSON object element username, and an integer from the JSON element result.

Figure 9: Extracted fields in the Results tab.
Figure 9: Extracted fields in the Results tab.

Summarizing the results based on these fields can be done with the following DQL:

fetch logs
| parse content, "LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)"
| filter username == "james" and result == 0
| summarize by: src_ip, count()

Imagine that you don’t have DPL available and need to filter all log records based on only searching for string values: the result would contain a lot of false positives (which only add “noise” to time-critical investigations). This is especially true with more complex log records containing nested JSON objects. The above example was oversimplified intentionally, but considering complex log records like AWS CloudTrail log records, having precise access to data in a specific node of an object makes a huge difference.

Summary

When performing security investigations or threat-hunting activities, it’s important to have precision in place to get reliable results. Historical data needs to be available, and access to object details is required for precise answers. DPL Architect enables you to quickly create DPL patterns, speeding up the investigation flow and delivering faster results. With the possibility of using Dynatrace-provided patterns for selected technology stacks, investigators can deliver answers even faster!

Check out the following video, where Andreas Grabner and I teamed up for a new episode of Dynatrace Observability Clinic. In this video, we dig deeper into the topic of extracting data via DPL, including a live demonstration of DPL Architect.

The post Speed up your security investigations with DPL Architect appeared first on Dynatrace news.

]]>
https://www.dynatrace.com/news/blog/speed-up-your-security-investigations-with-dpl-architect/feed/ 0