Trusted vulnerability data empowers teams to make faster and clearer security decisions. That’s why we’re elevating the Dynatrace Vulnerabilities app with the Dynatrace Vulnerability feed: a new, native source of vulnerability intelligence that’s more accurate, transparent, and threat-aware, helping maintain strong coverage of critical risks. With curated inputs, stronger sourcing, and deeper integration across the Dynatrace platform, teams can prioritize what matters most—and act with confidence.

A native vulnerability feed: Agile, precise, and focused on real customer risk

We recognize how quickly noise and alert fatigue can slow vulnerability management. When teams face a high volume of security findings, they require accurate, timely, and reliable threat intelligence to address potential breaches and optimize remediation time, resources, and costs. That’s why Dynatrace is upgrading the Vulnerabilities app to use the Dynatrace vulnerability feed—a new, internally curated source of vulnerability data that replaces the previously used external feed. It delivers more accurate, timely, transparent, and threat-aware vulnerability information, tightly integrated with innovations from Dynatrace security researchers, already recognized through a European patent.

The Dynatrace Vulnerability feed is built on multiple reputable sources, including OSV.dev, GitHub, the National Vulnerability Database (NVD), and vendor advisories, providing comprehensive coverage of vulnerabilities. These insights are further curated and enriched by Dynatrace’s own findings and internal security research.

Key benefits for customers

The new Dynatrace Vulnerability feed is included in the Vulnerabilities app at no additional cost and offers:

• Trustworthy, high-quality, curated vulnerability data

• Clearer and more consistent vulnerability descriptions and remediation guidance
• Strong coverage of critical and high-severity vulnerabilities
• Improved accuracy for certain findings compared to the previous feed
• Better long-term agility: Dynatrace owns and evolves the feed based on customer needs

• Deeper integration with the Dynatrace platform.

Coverage and parity with the previous vulnerability feed

The Dynatrace Vulnerability feed provides full parity with the previously used feed for critical and high-severity vulnerabilities in customer environments. For medium- and low-severity vulnerabilities, we ensure over 90% coverage in customer environments compared to the previous feed, while also adding extra vulnerabilities that were not previously included. Certain medium- and low-criticality vulnerabilities, as well as those without a CVE number that exist solely in the previous feed, will be marked as deprecated (Figure 1).

We recognize that coverage is a key topic for organizations when it comes to vulnerabilities. The Dynatrace Vulnerability feed is designed to be dynamic, keeping pace with the large volume of newly reported vulnerabilities. This agility ensures timely updates to meet customer-specific coverage needs, reducing blind spots and protecting against emerging threats. This flexibility is one of the key reasons we decided to introduce our own feed.

What changes for existing vulnerabilities?

From an access and viewing standpoint, nothing changes for existing vulnerabilities. You can continue accessing vulnerability insights from the existing Vulnerabilities app, ensuring a seamless experience with minimal disruption.

Existing vulnerabilities from our previous feed will retain their CVE numbers, but the IDs will be replaced with the Dynatrace Vulnerability IDs (Figure 2). Vulnerabilities from the previous feed without a CVE number will be automatically marked as resolved.

If your organization drives vulnerability remediation through an IT Service Management (ITSM) tool like ServiceNow, your existing tickets remain unchanged regarding CVEs and Dynatrace IDs. Tickets linked to vulnerabilities outside the Dynatrace Vulnerability feed will still function, but when accessed, the vulnerabilities might be deprecated, have updated severity, or have an adjusted score.

Availability for SaaS vs. Managed: What You Need to Do

The Dynatrace Vulnerability feed is available in all Dynatrace SaaS environments starting with version 1.334. If you’re on Dynatrace SaaS, this change will happen automatically when version 1.334 is rolled out in March 2026.

If you’re on Dynatrace Managed, an update to version 1.334 (or later) is required to use the Dynatrace Vulnerability feed. Be sure to update to version 1.334 or later before April 1, 2027; otherwise, you’ll no longer be able to analyze vulnerabilities.

Not a Dynatrace customer yet? Try the Dynatrace Vulnerabilities experience in our Playground and see how threat-aware, curated vulnerability intelligence helps you cut through noise and focus on real risk.

The post Introducing the Dynatrace Vulnerability feed: Accurate, transparent, and threat-aware appeared first on Dynatrace news.

With advanced filtering, deduplication, and visualization, teams can reduce noise, compare results across tools, and integrate findings directly into dashboards and workflows, thereby accelerating remediation and improving their overall security posture.

The challenge of fragmented vulnerability data

As organizations adopt more security tools and operate across hybrid and cloud native environments, vulnerability data becomes increasingly fragmented. Security teams struggle to correlate findings, assess real impact, and prioritize remediation across disparate sources. Existing solutions often provide siloed views, lack integrations, have limited filtering and analytics capabilities, forcing teams into manual triage and slowing response time. Such complexity makes it difficult to gain meaningful insights, scale vulnerability management, automate workflows, and consistently reduce risk.

Empower security champions with unified findings and advanced filtering

Until recently, the Vulnerabilities app main view focused on Prioritization, featuring a display of stateful security problems, for example, specific CVEs or code issues affecting multiple process groups, along with a reachability assessment and the history of the problem.

Findings are the atomic building blocks behind these problems: individual security results tied to a specific issue, moment in time, and affected object.

With the new Findings tab, Dynatrace consolidates all vulnerability findings—whether from Dynatrace vulnerability detection or integrated tools—into a single, unified view. This allows analytics-ready data, precise filtering, segmentation, and cross-tool comparison, while empowering deeper insights and automation, all serving to accelerate remediation and improve your overall security posture.

Drill-downs and analytics based on atomic data

The new Vulnerability Findings feature delivers a unified, granular view of all security findings from Dynatrace and integrated tools. It simplifies vulnerability management by consolidating fragmented data, enabling faster root cause analysis, and supporting automation for remediation workflows. This improvement provides holistic visibility across environments, atomic-level detail for each finding, and analytics-ready data for advanced automation and security insights. Key capabilities include powerful filtering (by CVE, score, vulnerable component, severity, and remediation availability), deduplication to reduce noise, and interactive visual timelines for trend analysis.

Use cases

• Investigation: Security teams drilling down into specific findings for detailed investigation, viewing findings affecting a specific component, object or adhering to a specific CVE
• Comparison: Comparing vulnerabilities across multiple tools, affected objects and environments
• Automation: Quickly building dashboards and workflows for proactive monitoring and automated response: the Findings tab is Grail native and therefore easily integrates with other Dynatrace apps
• Targeted remediation: Segmenting findings by metadata (e.g., leveraging Primary Grail Fields like Kubernetes namespace) for targeted remediation

Upcoming enhancements will introduce flexible grouping features to make it even easier to grasp the value of vulnerability findings. You’ll be able to group findings by CVE, vulnerable component, or affected object for faster analysis and prioritization.

Get started

These enhancements for Vulnerabilities are available now for Dynatrace Platform Subscription (DPS) customers. If you’re new to Dynatrace and want to experience these powerful capabilities, you can try them out on the Dynatrace Playground.

Existing customers will automatically see these enhancements. Start exploring the Findings tab in Vulnerabilities today and gain a unified, granular view of your security findings. Reduce noise, prioritize risks with confidence, and accelerate remediation using powerful filtering and analytics-ready data.

Related resources

• Integrate vulnerability insights across Dynatrace and external apps
• Documentation: Vulnerabilities App
• Prioritize vulnerabilities based on the CISA Known Exploited Vulnerabilities Catalog
• Discover the new Dynatrace Runtime Vulnerability Analytics experience

The post Introducing findings in the Vulnerabilities app: Unified, granular insights for smarter security appeared first on Dynatrace news.

Dynatrace has enhanced its runtime vulnerability analytics by integrating data from the Cybersecurity & Infrastructure Security Agency’s (CISA) Known Exploited Vulnerabilities (KEV) catalog, complete with critical remediation due dates. This powerful addition allows organizations to prioritize actively exploited vulnerabilities, align with federal security standards, and focus remediation efforts where they matter most—on threats that attackers are weaponizing right now.

The challenge of vulnerability prioritization

In today’s threat landscape, security teams face an overwhelming challenge in determining which vulnerabilities pose the greatest risk to their organization. With thousands of new vulnerabilities discovered each year, traditional scoring systems like the Common Vulnerability Scoring System (CVSS) often fall short of providing the real-world context needed for effective prioritization.

The reality is that not all vulnerabilities are created equal. While a vulnerability might receive a high CVSS score based on its theoretical impact, it might never be exploited in the wild. Conversely, some vulnerabilities with moderate scores are favorite tools of threat actors and are often involved in widespread attacks and significant business impact.

This disconnect between theoretical risks and actual threats creates several challenges for organizations:

• Resource allocation inefficiencies: Teams spend valuable time patching vulnerabilities that might never be exploited, while more dangerous threats remain unaddressed.
• Alert fatigue: Security professionals become overwhelmed by the sheer volume of vulnerability notifications, which leads to critical issues being missed.
• Compliance gaps: Organizations struggle to align their security practices with federal standards and regulatory requirements.

Enhanced Vulnerability Management with CISA KEV

Dynatrace already provides a Davis Security Score, an enhanced risk-calculation score that’s based on the industry-standard CVSS. Davis® AI provides this precise risk-assessment score by considering additional parameters such as public internet exposure and whether or not data assets are reachable from an affected entity.

Now, Dynatrace has further enhanced its runtime vulnerability analytics by integrating data from the CISA KEV catalog, including the critical due dates for remediation. This addition brings a new layer of actionable intelligence to vulnerability management by unlocking the ability to prioritize actively exploited vulnerabilities.

Unlike general vulnerability databases, KEV is based on real-world exploitation data and expert analysis. Each entry is carefully vetted and includes a due date by which federal agencies must remediate the issue. This due date is not arbitrary—it reflects the urgency of the threat and the need for swift action. By aligning with KEV, Dynatrace helps organizations stay ahead of attackers by focusing on vulnerabilities that are not just theoretical but actively weaponized.

The KEV catalog is especially valuable because it’s curated by cybersecurity experts at CISA using intelligence from government, industry, and open source reporting. Customers integrating KEV into their security workflows can align with federal standards, improve risk-based prioritization, and reduce exposure to high-impact attacks.

KEV filtering and prioritization

Once you’ve turned on runtime vulnerability analytics in Dynatrace, you can filter KEV vulnerabilities in the Vulnerabilities app. The resulting view is sorted based on the remediation due date, starting with the most urgent vulnerability.

Expanding KEV support

CISA KEV data is available directly within the Vulnerabilities app, giving your teams immediate visibility into the known exploited vulnerabilities that affect their environment and the remediation date by which they must be resolved. In the future, Dynatrace will expand support for KEV data to vulnerability dashboards and automation workflows so you can visualize KEV-related risks in your custom dashboards or assign remediation tasks based on KEV due dates or exploit status.

Try KEV filtering today

This integration helps organizations align with federal security standards, improve risk-based prioritization, and reduce exposure to high-impact attacks. It’s a practical way to stay ahead of cybersecurity threats by focusing on vulnerabilities that are actively exploited.

The post Prioritize vulnerabilities based on the CISA Known Exploited Vulnerabilities Catalog appeared first on Dynatrace news.

Python is a popular programming language with a clear and readable syntax. Python’s versatility allows it to be applied in various fields, from web development to data science. Detecting vulnerabilities in Python is crucial due to its widespread use in critical applications, which makes it a prime target for attackers. Additionally, Python projects often rely on third-party libraries, which can introduce risks if not properly monitored. Proactive vulnerability monitoring ensures compliance with security standards and helps prevent costly security incidents.

Real-time Python vulnerability detection in production

With Dynatrace Runtime Vulnerability Analytics extended to monitor Python, organizations can use the newest Python libraries while making sure that all code running in production and pre-production environments is subject to continuous, stringent security monitoring. Newly published CVEs are detected immediately, allowing security teams and champions to quickly analyze the actual risk, triage the most impactful vulnerabilities, and have them remediated. RVA analyzes if an application uses vulnerable Python libraries at runtime or a vulnerable Python runtime to execute the application code.

Similar to other monitored technologies, we provide full visibility into all affected processes, including related services, applications, and hosts, as well as Kubernetes workloads, nodes, and clusters. Mitigators can quickly prioritize vulnerabilities based on network exposure and understand which data is at risk and how easily vulnerabilities can be exploited by an attacker.

Python vulnerability monitoring is easily set up for all hosts monitored by Dynatrace OneAgent®. Dynatrace detects if you’re using vulnerable third-party libraries or runtimes to execute your code. Activate Python monitoring in the security settings of the Vulnerabilities app. The app allows you to filter, sort, and connect vulnerabilities to specific remediation tickets. Python vulnerability monitoring is also available for Dynatrace Managed customers.

Get started

Activate Python vulnerability monitoring in Vulnerabilities app settings or explore it in the Dynatrace Playground.

The post Dynatrace launches Python Vulnerability Monitoring for enhanced customer security appeared first on Dynatrace news.

Key benefits of Runtime Vulnerability Analytics

Managing application vulnerabilities is no small feat. Traditional tools often overload you with data, making it challenging to identify which vulnerabilities actually put your environment at risk. Dynatrace’s unique ability to understand which assets in your production environment are actively exposed—such as those vulnerable to internet-based threats or linked to sensitive data sources—allows you to immediately focus on what matters most.

The updated platform does more than detect vulnerabilities— it gives you an in-depth understanding of their potential impact. For example:

• Exploitation awareness: Identify vulnerabilities based on whether they’re exposed to critical assets or have exploits circulating in the wild.
• Real-world context: Determine if vulnerabilities are linked to internet-facing systems or databases to help you prioritize the vulnerabilities that pose the greatest risk.
• Streamlined prioritization: Assess threats based on their real impact on your environment, not just abstract risk scores. By focusing on actionable intelligence, you can reduce noise and focus on what’s important.

What’s new in this version?

Unified vulnerabilities view

The new Dynatrace platform consolidates third-party and code-level vulnerabilities into a single, intuitive view. Instead of switching between multiple views, you have a comprehensive overview of your environment’s vulnerabilities in one place.

Advanced filtering capabilities

With the newly added filtering field, you can tailor how you search for vulnerabilities:

• Combine search criteria for complex queries, such as finding vulnerabilities connected to data assets but not exposed to the internet.
• To filter findings efficiently, use numerical thresholds like DSS (Dynatrace Security Score) or CVSS (Common Vulnerability Scoring System).
• Search full vulnerability descriptions for pinpoint accuracy.
• For instance, you can quickly locate vulnerabilities exposed to the web with a DSS score higher than 8, while CVSS scores lower than 10 allow you to focus on risks that require immediate attention.

View segmentation

The Vulnerabilities app utilizes a cross-platform segmentation feature that helps you focus on specific areas of your environment:

• Slice environments into categories like process groups, applications, or even individual services.
• Create custom segments based on attributes like vulnerability type or Davis® AI assessment.
• For example, you might create a segment that tracks vulnerabilities in your payment processing system separately from general infrastructure assets.

Davis Security Score adaptability

The Davis^® Security Score (DSS) now adapts to the segments you’re viewing. If a selected segment contains only medium-critical entities, the vulnerability’s score will reflect that, ensuring more precise and context-relevant prioritization.

Why these features matter

Imagine your web application uses a vulnerable library that’s directly exposed to the internet and connected to data assets. The unified prioritization view, combined with advanced filtering and segmentation, helps you identify this issue quickly, assess its risk level, and prioritize fixing it—all before it can be exploited.

Secure your applications now

The new Dynatrace Vulnerabilities app simplifies complexity and helps you tackle threats head-on. Log into your Dynatrace account today to unlock the full potential of your application’s security monitoring.

Note: Switching the Vulnerabilities app to a Grail-native backend requires updated permissions for users. Please see the instructions in Dynatrace Documentation.

Not a Dynatrace customer yet? Explore these new capabilities in the Dynatrace Playground.

The post Discover the new Dynatrace Runtime Vulnerability Analytics experience appeared first on Dynatrace news.

As teams conduct forensic investigations into the MOVEit Transfer vulnerability, many are finding they can’t conclusively determine their exploitation status. This uncertainty is because they may not have collected enough logs, or the logs don’t contain enough information. When working with logs originating from an out-of-the-box software such as Progress MOVEit Transfer, we can’t do much. What the software has been designed to log is what we get. But observability data (traces) can fill in the blanks to reveal useful evidence of possible exploitation, as proved by our analysis of the MoveIT vulnerability using Dynatrace.

What is the MOVEit vulnerability?

The MOVEit vulnerability is a critical SQL injection flaw in the web application MOVEit Transfer, which MITRE published as CVE-2023-34362 on May 31, 2023. When exploiting the vulnerability, attackers can gain remote code injection capabilities in the MOVEit server and modify or steal sensitive data from its database. On June 6, 2023, the ransomware gang Cl0p claimed responsibility for exploiting the MOVEit Transfer vulnerability starting on May 27, 2023, during the U.S. Memorial Day holiday. As a zero-day vulnerability, however, threat actors could have been exploiting the vulnerability since 2021, when the vulnerability was introduced into the software. Victims include large financial institutions, government agencies, and other critical service providers.

As soon as MITRE published the CVEs and the bad actors launched their exploits in the wild, the infosec community got busy reverse engineering malware and investigating attacker tactics. Progress Software, the company behind MOVEit softwareTransfer, has published an excellent overview, as have Huntress Labs, Mandiant, TrustedSec, Horizon3.ai and others.

Using Dynatrace to investigate the MOVEit Transfer vulnerability

Investigating the MOVEit vulnerability is a perfect use case for Dynatrace: having all your observability and security data immediately available to investigate a potential compromise, combined with indexless, schema-on-read storage and excellent processing speed, with one language (Dynatrace query language—DQL) to query them all.

To see how Dynatrace investigates the MOVEit vulnerability, we set up a proof of concept (PoC) to explore the exploit and its impact in a sandbox environment. The question was: Using Dynatrace Grail and DQL, what kind of attacker activity can we discover in the logs using an out-of-the-box logging setup both for the MoveIT vulnerability and the IIS logs of the server running the software?

As we started experimenting with the PoC and looking for the indicators of compromise (IoCs) in the IIS logs published by the community, the sky seemed clear. Running blazing-fast queries on historical logs stored in Grail was pretty straightforward. Lucky, because the vulnerability reportedly dates back to 2021. Because IIS logs contain IP addresses, client requests, and server response codes, querying them seemed quite promising to find suspicious indicators like certain IP addresses, strings and requests reported by the community.

For example, the LEMURLOOR web shell Cl0p used during the Memorial Day attacks used a backdoor called human2.aspx. This backdoor enabled attackers to access MOVEit data and users, download files, insert an administrative user, which enabled attackers to bypass credentials. The following query looks for the string human2.aspx in logs to detect if the attacker has added a webshell to exploit the MOVEit software.

The MOVEit vulnerability log quality wall

As the research progressed, however, we hit the log quality wall.

We found that the MOVEit default logs contained no useful information on the attacker’s activity.

The problem with the IIS server access logs was the headers. IIS access logs often log some headers but not all headers. So we’re likely to miss important evidence that would make the investigation and attack detection much faster.

And unfortunately, payloads containing valuable information on the exploits are not logged at all. Payloads would be nice, although the logs would get too crowded and full of privacy issues. But a security analyst can always dream…

These deficits left us unsure about what the attacker had done, their methods, and what data they compromised.

Using observability data for indicators of compromise

The disappointment with log quality sent us looking for more data to dig into. A good source for additional information is observability data. We can use metrics to detect anomalies, events to find unusual activities, and traces, which provide detailed insights into system activities. In Dynatrace, we can use one query language, DQL, to query all these different data sources.

For example, to see if the attackers installed a web shell, we can use DQL to find spans that contacted the endpoint human2.aspx.

In this query, we first fetch the spans (a span represents a single trace segment). Next, we filter them by the MOVEit service to improve the query performance. Finally, we filter by http.target that contains human2.aspx.

There are no results in this case, but this doesn’t mean there was no exploitation. There are other options to exploit the vulnerability. And since this IOC is now well-known, attackers will likely do it differently. You can normally find this information easily in logs (such as  IIS logs). But when you can get the information from spans, results get much more interesting.

Spans reveal more about MOVEit vulnerability exploit activity

Using spans, for example, we can find evidence of MOVEit vulnerability exploit activity in database queries. The exploit works by modifying the access tokens using SQL injection (SQLI). By using DQL to query spans, we can see if there were updates to the respective tables.

This is a simple query to check for updates on the hostpermits table, which is necessary to use the created access token from a remote host. The query is quite simple, but it might miss some matches. For example, if the database name is prepended or the table name has backticks around it. Let’s improve on the query to get better results and look for additional tables at the same time:

The improved query matches a wider variety of options and we can see that it found several SQL injections. The parse section of the query performs the following logic:

• DATA: matches any data, to match the beginning of the SQL query
• update: matches a string literal to find update statements
• BLANK: after the update keyword, there needs to be a space, but it could be more than one
• ('moveittransfer.'|'`'|''): matches either movittransfer (if the database name is prepended), a backtick (table names can be surrounded by backticks), or nothing
• ('hostpermits' | 'userexternaltokens'| 'trustedexternaltokenproviders'): matches against different table names

Logs, traces, spans – you need them all for investigations like the MOVEit vulnerability

It’s not possible to create logging from scratch for out-of-the-box software. We need to work with what we can get. Existing logs might not contain enough information to investigate the incident. Teams may have disabled useful logging by default, and when they discover the incident, it’s too late to configure logging. Therefore, looking at observability data jointly with logs can be a game changer.

Our investigation of the MOVEit Transit vulnerability proved that querying spans can give us valuable missing pieces to detect what the attackers were up to.

Interested in trying out these queries in your own environment? If you’re already a Dynatrace customer, these queries are available in a Dynatrace Notebooks template. Just ask your Dynatrace account rep.

The post MOVEit vulnerability: Observability context fills log data gaps for MOVEit Transfer vulnerability appeared first on Dynatrace news.

Log forensics—investigating security incidents based on log data—has become more challenging as organizations adopt cloud-native technologies. Organizations are increasingly turning to these cloud environments to stay competitive, remain agile, and grow.

But as organizations rely more on cloud environments, data and complexity have proliferated. Teams struggle to maintain control of and gain visibility into all the applications, microservices and data dependencies these environments generate. Without visibility, application performance and security are easily compromised.

As a result, teams are turning to technologies such as observability to understand events in their cloud environments. Moreover, they have come to recognize that they need to understand data in context. But most observability technologies today provide information in silos. Without unifying these silos, teams miss critical context that can lead to blind spots or application problems—problems that compound when there’s a need to investigate security events.

Modern observability enables log forensics

Dynatrace is a software intelligence platform that provides deep visibility into and understanding of applications and infrastructure. It started as an observability platform; over time, it has expanded to provide real user monitoring, business analytics, and security insights. The recent innovation around log storage, processing, and analysis—Grail—makes Dynatrace a great solution for security use cases such as threat hunting and investigating the who-what-when-where-why-how of an incident.

Grail is a data lakehouse that retains data context without requiring upfront categorization of that data. Unifying data in Grail brings critical security capabilities to bear as teams seek to understand malicious events.

Grail enables organizations to find and analyze security events in the context of their broader cloud environments. Moreover, with capabilities such as log forensics—the analysis of log data to identify when a security-related event occurred—organizations can explore historical application data in its full context.

Grail makes it easy to query historical data without data rehydration or indexing, re-indexing, and up-front schema management. This accessibility gives users quick and precise results about when malicious activity occurred, when reconnaissance was first seen in the systems, what was attempted, and if the attackers were successful.

Demo: “Ludo Clinic” uses log forensics to discover and investigate attacks using Grail

Imagine working as a security analyst for a respectable medical institution called Ludo Clinic. As Ludo Clinic started using Dynatrace, the platform’s Runtime Application Protection feature detected a SQL-injection (SQLI) attack. Thanks to the details provided by the code-level vulnerability functionality, the developers knew where in the code the exploited vulnerability was and were able to address and patch it quickly.

The task now is to investigate whether the system experienced any suspicious activity before the attack so we can determine if any other systems are affected. The good news is there are metrics available a few days before your team detected the attack, and you also ingested three months of application and access logs into Grail. Because these logs are ready for querying with no rehydration, the investigation can start immediately. The bad news is we don’t know exactly what to look for. “Find suspicious activity” can mean anything. So, we’ll start by exploring the data using the hints and context information we already detected with Dynatrace.

Hint one: Blocked SQL injection report details

Here’s the report from Dynatrace on the blocked SQL injection details on 10 February from the IP 104.132.226.34. This report shows details of the attack, such as the entry point, the vulnerability that was exploited, the IP address of the attacker, and so on.

Hint two: Failed logins spike

A quick look at the metrics dashboard dating back to 8 February shows a spike in failed logins metrics before your team detected the attack. Indeed, there’s a spike on 8 February.

It would make sense to see if there’s any activity from that IP before we set up monitoring. Has the attacker been doing reconnaissance from that same IP in our systems even before this? If yes, how? Did they try something else during those three months? Were they successful?

Notebooks, DQL, and DPL: Tools of the Grail log forensics trade

Now that we have some clues about where to look for suspicious activity, we’ll dig into the logs using Notebooks, DQL, and DPL.

Dynatrace Notebooks

Dynatrace Notebooks is a collaborative data exploration feature that operates on data stored in Grail for ad-hoc exploratory analytics. Notebooks enable cross-functional teams, such as IT, development, security, and business analysts, to build, evaluate, and share insights for exploratory analytics using code, text, and rich media. The ability to build insights from the same data using the expertise of different roles helps organizations truly understand everything their data has to say.

Dynatrace Query Language (DQL)

Dynatrace Query Language (DQL) is a piped SQL-like query language, similar to Linux commands executed in sequence. You can look at the queries like a series of building blocks applied in an order you happen to need at this moment. Select fields, summarize, and count a value, apply more filters, select additional fields, extract data from a particular field, and so on. DQL is great for exploring and experimenting with data, which makes it a great ally in log forensics and security analytics.

The first query of our investigation uses DQL in Notebooks to fetch logs from Grail, filter the access log, and limit the result to 1000 records for initial exploration.

Dynatrace Pattern Language (DPL)

In our investigation, we’ll also use DPL. DPL stands for Dynatrace Pattern Language, a parsing language that also consists of intuitive building blocks that help to extract meaningful fields from data on read. That means there is no need to manage indexes and rehydrate archived data; simply specify an ad hoc schema using DPL as part of the query.

What’s more, with DPL, the parsed results return typed fields, so you can be sure that a timestamp is a timestamp and an IP address is an IP address, not some random octets separated by a dot like 320.255.255.586. Working with typed data means excellent quality and precision for investigation results because you can run type-specific queries like calendar operations, calculations on numeric data, working with JSON objects, and so on. Working with typed data means excellent quality and precision for investigation results, as you can run type-specific queries like calendar operations, calculations on numeric data, working with JSON objects, and so on.

Log forensics: Querying the access log

Remember: our task is to investigate whether any other systems are affected. The first step is to query whether the IP address where the SQLI attack came from has been used before. Can we see it in the web application access log months prior to the attack?

The query result shows there is no activity from that IP earlier than records on 10 February, the day Dynatrace detected the SQL injection attack. This means that the attack appeared “out of the blue,” and it is likely the attackers were using other IP addresses to do reconnaissance on our systems.

Because we can’t find the attacker by the IP address, let’s look at abnormalities in login behavior because there is a chance they’ll be related to reconnaissance. This means we’ll investigate the spike in failed logins we saw earlier in the metrics graph. Are there any other failed login spikes three months prior to the attack? Where do the failed logins originate from?

We can see that a failed login attempt takes users to a specific URL:

/ludo-clinic/login?authenticationFailure=true

So, let’s see if and how often this URL appears in the logs by adding the following filter to the query.

| filter contains(content, "/ludo-clinic/login?authenticationFailure=true") 
| limit 10000

Indeed, the query gives us 6531 records containing a failed login URL:

Making sense of the access log

For the next stage of our investigation, let’s make more sense of these ~6,300 records and find out how many unique IP addresses were the origin of failed login attempts. The hypothesis is: some of the IP addresses stand out when it comes to the number of login failures. This means we first need to extract the IP address to run this aggregation.

We can utilize the schema-on-read functionality, that is, extract only the fields we need for a specific query. Taking a closer look at the content field of the access log, we can see a traditional HTTP access log: clientIP, timestamp, requestURL, HTTP response code, and so on.

Notice that the timestamp field (the ingest timestamp) is similar for all log records (15/05/2023 14:09:48). This is because Ludo ingested the historical log records in bulk. To analyze the event time, we need to extract the timestamp from the content field as event time. To count IP addresses, we also need the IP address.

Quick ad-hoc parsing to aggregate login failures

To parse out data (timestamp and IP) from the content field, we’ll select the content field and select Extract fields to open the DPL Architect. To retrieve the timestamp and clientIP, we’ll replace the default DPL pattern with the following:

IPADDR:client_ip LD HTTPDATE:event_time

This matches and extracts the timestamp and the IP address from the content field and gives them a name (event_data and client_ip) and leaves the rest of the pattern unmatched, as we don’t need it for the following query. Clicking Insert pattern brings us back to the query view, adding a parse command to the newly created pattern.

Now with the extracted IP address, we can proceed with queries and use the summarize command to count the number of failed logins per unique IP address to see if there were any failed logins originating from a specific IP. Sorting the result set based on the number of failed logins in descending order gives us the largest outliers.

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:client_ip LD HTTPDATE:event_time" 
| fields event_time, client_ip, content 
| summarize total=count(), failed=countIf(contains(content, "/ludo-clinic/login?authenticationFailure=true")), by:client_ip 
| sort failed desc

This pays off! The results reveal that a significant portion of logins (181,774) and failed logins (6161) originate from the IP address: 172.31.24.11. This seems interesting and is worth taking a closer look.

Timing of login failures

Next in our log forensics journey, let’s see when these failed logins from that particular IP address occurred to get more information on the potential reconnaissance activity. Did the requests all occur within a short period or regularly across a longer period?

Because we’re interested in the behavior of a specific IP and investigating the reasons behind failed logins, let’s also extract the session ID from the log line. As the session ID is the only field that occurs both in the access and application weblogs, it will be also useful later when we need to join the two for investigating affected users.

We already extracted the IP and included the timestamp (HTTPDATE). We will now extend the pattern and skip the part of the record we don’t need by not naming the three double-quoted strings (DQS). Finally, we’re extracting the last field that contains the session ID.

IPADDR:client_ip LD HTTPDATE:event_time LD DQS LD DQS LD DQS SPACE LD:session_id

When we select Insert pattern, we again get a parse command populated with the DPL pattern we just created.

Focusing on the suspicious IP

Next, let’s select only the fields we’re interested in and then aggregate fields. These actions reveal more about the extended activity that involves the suspicious IP address responsible for many of the failed logins.

| fields time, client_ip, session_id, content

Filtering the attacker IP and sorting the fields based on the timestamp we just parsed out, it appears this IP address was first seen on 24 December 2022. We now know the start of the suspicious activity. For malicious actors, it is quite common to act during the holiday period.

fetch logs  
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, " IPADDR:client_ip LD HTTPDATE:event_time LD DQS LD DQS LD DQS SPACE LD:session_id" 
| fields event_time, client_ip, session_id, content 
| filter contains(content,"172.31.24.11") 
| sort event_time asc 
| limit 300000

Find the suspicious activity pattern across time

To see the activity pattern of this suspicious IP across time, let’s count the number of failed logins in one-hour time intervals.

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:ip LD HTTPDATE:time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
| fields time, ip, sessionID, content 
| filter contains(content,"172.31.24.11") 
| summarize failed=countIf(contains(content, "/ludo-clinic/login?authenticationFailure=true")), by:bin(time, 1h)

It appears as though failed logins from this IP appear to follow a very regular pattern: 24 failed attempts every hour. Looks like this activity is automated and most probably refers to a dictionary attack: regular (automated) attempts from the attacker to try out different usernames and passwords, mostly with failed results.

But to escape the clinic’s countermeasures (failed login attempts velocity check), the attacker also conducts a successful login every now and then. If we count all activity from that IP address (not just the failed logins but successful attempts as well), the results are again very symmetrical:

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:ip LD HTTPDATE:time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
| fields time, ip, sessionID, content 
| filter toString(ip) == "172.31.24.11" 
| summarize count=count(), by:bin(time, 1h)

Identify targeted users

Next, it would be useful to know which users the attacker has targeted and whether any attempts have been successful. Let’s aggregate the activity from this IP using sessionIDs:

fetch logs 
| filter contains(log.source, "ludo-clinic-access.log") 
| parse content, "IPADDR:ip LD HTTPDATE:event_time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
| fields timestamp, event_time, ip, sessionID, content 
| filter contains(content, "172.31.24.11") 
| summarize count=count(), 
            by:{sessionID 
               } 
| sort count desc 
| limit 10000

The result is again peculiar, suggesting automated activity: 59 log lines per session.

Next log forensics dataset: The web application log

Next, let’s see what was happening based on the web application log, using data from what was going on during those sessions that originated from the suspicious IP address we discovered from the access log dataset.

First, to familiarize ourselves with the content of the webapp log, let’s run a basic query to see what the content field of the web application log looks like:

We can see a timestamp, log severity, traces and spans, a session ID, result, and username. There are plenty of interesting fields to play with, so the next step is to parse the content into fields that are ready for querying. We can extract the fields using the DPL Architect. The following DPL pattern extracts the event time session ID, result, and username from the webapp log.

'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS)

Inserting the pattern, this is what the query looks like when parsing out session IDs and usernames from the web application log.

fetch logs, from:-300d   
| filter contains(log.source, "ludo-clinic-webapp.log") 
| parse content, "'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS) " 
| limit 10000 
| fields event_time, sessionIdApp, result_text, username

Filter out records from the authentication provider

To see which users were targeted and how successful the attacker was, we will continue working with only those web application records that contain authentication responses. First, we filter out the records that originate from the authentication provider, then we skip the responses we’re not interested in:

| filter contains(content, "CustomAuthenticationProvider")  
  AND NOT contains(content, "Starting findUsersByUsernameAndPassword") // we want to see only auth response log records 
  AND NOT contains(content, "retrieved matching list")

The full query now looks like this and returns the following results:

fetch logs  
| filter contains(log.source, "ludo-clinic-webapp.log") 
| filter contains(content, "CustomAuthenticationProvider")  
  AND NOT contains(content, "Starting findUsersByUsernameAndPassword") // we want to see only auth response log records 
  AND NOT contains(content, "retrieved matching list") 
| parse content, "'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS) " 
| limit 10000 
| fields event_time, sessionIdApp, result_text, username

Review the user sessions that originate from the attacker

Next, to see which user sessions in the webapp log originated from the attacker’s activity, we use a lookup query to join aggregated sessions from the attacker IP address we discovered in the access log with sessions in the webapp log. In short, we will see what was happening during the suspicious sessions according to the webapp log.

fetch logs  
| filter contains(log.source, "ludo-clinic-webapp.log") 
| fields content 
| filter contains(content, "CustomAuthenticationProvider")  
  AND NOT contains(content, "Starting findUsersByUsernameAndPassword") // we want to see only auth response log records 
  AND NOT contains(content, "retrieved matching list") 

| limit 10000 
| parse content, "'[' TIMESTAMP('dd/MMM/yyyy:HH:mm:ss.S'):event_time LD ' - ' LD:sessionIdApp ' ' LD:result_text ': ' LD:username (';' | EOS) " 
| lookup [fetch logs, from:-300d 
                | filter contains(log.source, "ludo-clinic-access.log") 
                | filter contains(content,"172.31.24.11") 
                | parse content, "IPADDR:ip LD HTTPDATE:time LD DQS LD DQS LD DQS SPACE LD:sessionID EOS" 
                | filter isNotNull(sessionID) 
                | limit 100000 
                | summarize accesscount=count(), by:{sessionID} 
                | fields sessionID], sourceField:sessionIdApp, lookupField:sessionID 

| fieldsRemove content

The result shows the attacker has achieved both successful authentications as well as failed authentications. Finally, we see which usernames the attack targeted the most by looking for the response “No users found requested username.” The system returns this value when it receives a non-existent user or a wrong password. By aggregating the result based on unique usernames, we get a list of the most (unsuccessfully) targeted users.

| filter result_text == "No users found requested username" 
| summarize count(), by:{username} 
| sort `count()`desc

These results are fascinating – we can see five usernames that the attacker continuously entered and received failed authentication results. We can also see SQL commands instead of regular usernames.

Determine successfully targeted users

Next question: Did they achieve anything besides ‘No users found’ when targeting these users? Let’s have a look by excluding the “No users found requested username” response and concentrating on those five users from the last query result, and adding the following line to the query:

|  filter not matchesPhrase (result_text, "No users found requested username") and in (username, "arnie", "herman", "krzysztofs", "bernice", "sherry")

Indeed, we see a lot of “successfully authenticated” responses in the result text field. This confirms the attackers were successfully conducting a dictionary attack: trying out several usernames and passwords to authenticate as real users of Ludo Clinic. When counting the number of successful authentications per these five users, the results are quite similar:

|  filter not matchesPhrase (result_text, "No users found requested username") and in (username, "arnie", "herman", "krzysztofs", "bernice", "sherry") 
| summarize count(), by:{username} 
| sort `count()`desc

Investigation results from log forensics and metrics with Dynatrace

As a result of Dynatrace detecting a SQL vulnerability, anomalies in metrics, and subsequently running forensic queries on three months of logs prior to the attack, we’ve been able to construct the following timeline:

• As Ludo Clinic started using Dynatrace, they were able to observe a spike in metrics capturing failed logins on 8 February
• The system detected and blocked a SQL injection attack on 10 Feb (Fri)
• There was no other activity from that IP in the access logs (the logs reach back three months)
• When aggregating failed login activity, we discovered the following details: the IP address 172.31.24.11 stands out from the rest, counting to 6161 failed logins based on the access log during the past three months
• This IP was first seen in the logs on 24 December 2022 (the earliest timestamp for this set of logs is 20 November 2022)
• The sessions contain identical activities during identical timeframes, which suggests the attacker was using automated tools
• Joining sessions from the access log to the application log reveal almost ten thousand records originating from the suspicious IP 172.31.24.11
• It looks like the attackers were attempting a dictionary attack because it targeted several users at regular intervals, resulting in failed as well as successful authentications.
• Users stafford, ray, orrel, doug and joby were targeted to discover their passwords. The attacker successfully authenticated 35-37 times per user.
• The attackers also entered SQL statements instead of usernames attempting SQL injection attacks.

The DQL and DPL advantage

For such investigations, DQL and DPL make it convenient to quickly investigate and query logs for security analytics use cases that require drawing broad conclusions from the data one minute and then zooming into the activities of a specific session the next. An interesting find inspires the analyst to parse out yet another field and run aggregations on this data. As historical data is always ready for querying, all hypotheses can be quickly verified or dismissed. A curious mind and the right log forensics tools (DQL and DPL) make a great combination for fighting evil.

The post Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail appeared first on Dynatrace news.