VMware Aria Operations for Logs (formerly known as vRealize Log Insight) is used across enterprises to collect logs and provide analytics. The company recently announced a high-severity vulnerability in an earlier version of the tool. Tracked as CVE-2023-34051 (CVSS score: 8.1), the Aria Operations for Logs vulnerability relates to a case of authentication bypass that could lead to remote code execution. Basically, this means that an unauthenticated malicious actor could inject files into the operating system of an impacted appliance, which can result in remote code execution.

James Horseman from http://Horizon3.ai and the Randori Attack Team have been credited for the discovery and reporting. They have made a PoC for the vulnerability available and published the relevant indicators of compromise (IoC).

This vulnerability is a so-called “patch bypass” for a set of critical flaws that were addressed by VMware earlier this year, also discovered by Horizon3.ai. In their report, they presented how an attacker could use three different CVEs to achieve remote code execution. Since the patch only blocks access to Thrift services by IP and does not fix the other CVEs in VMSA-2023-0001, all an attacker needs to do is spoof an IP address and use the published attack again.

In this blog post, we show how to discover the original attacks toward the Aria Operations for Logs vulnerability using Dynatrace and DQL by finding the IoC-s from the log records.

How to detect VMware Aria Operations Logs exploitation with Dynatrace

Since we’re talking about a patch bypass, we’re looking at the original attack vector described in this blog post. To exploit this vulnerability to gain remote code execution (RCE), the following steps have to be taken by the malicious actor:

1. Create a Thrift client to gain unauthenticated access to the Log Insight Thrift server.
2. Spoof the IP address of the known worker.
3. Create a malicious TAR file containing a directory traversal using a valid PAK file.
4. Using remotePakDownloadCommand, upload the malicious PAK file to /tmp/<filename>.pak.
5. Extract the file using pakUpgradeCommand. This writes the file to where it’s needed on the filesystem.

A technical deep dive about the attack can be found in this article, VMware vRealize Log Insight VMSA-2023-0001 Technical Deep Dive.

Discovering the attack with Dynatrace

To discover this attack, you have to go through the log files that Aria Operations for Logs creates. The Log Insight server stores relevant logs in the file /var/log/loginsight/runtime.log. This log file is used to track all runtime information related to Log Insight. To analyze these logs with Dynatrace and DQL, the logs need to be ingested into Dynatrace beforehand.

Forwarding logs away from the main system is beneficial in case of any kind of successful attack by a malicious actor: if an attacker decides to remove all evidence from the system (including log files), then a security investigator wouldn’t have traces left to analyze. Sending logs to Dynatrace at runtime safeguards the log files for later analysis in a remote and secure location.

Security-related log records of Aria Operations for Logs are structured in the following format:

[2023-10-25 11:28:29.709+0000] ["https-jsse-nio-443-exec-9"/10.153.234.136 DEBUG] [com.vmware.loginsight.web.actions.misc.LoginActionBean] [User login success: vIDM: SAM=myusername, Domain=vmware.com, UPN=myusername@vmware.com]

When a new remote PAK file is downloaded as a tarball, the following log lines are created:

[com.vmware.loginsight.daemon.commands.SystemCommands] [PAK download initiated by node f2449ed5-11ee-45fd-a0a0-6225a33a8ac6] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Deleting existing PAK file: /tmp/exploit.pak] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak] 
[com.vmware.loginstght.daemon.commands.SystemCommands] [Max allowed pack size is 1505916450 bytes] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Current downloaded size is 21910]
[com.vmware.loginsight.daemon.commands.SystemCommands] [Total downloaded size is 21910]

From these log lines, the best candidate to look for in the logs is the third populated log, Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak].

Now, let’s fire up the DPL Architect and create a suitable pattern for this line:

With the created DPL pattern, you can extract the timestamp field ts, the logger class name as class that populated the log lines and the action that was written to the log record. These extracted fields give you the option to filter out the relevant records:

However, there’s another goal to keep in mind: you need all the log records that contain the Downloading statement and URL from where the tarball was loaded and the destination path in /tmp/.

As the blog post states: “The URL and filename will likely be different. However, the filename will always have the format /tmp/<filename>.pak. Also, note that this log may be legitimate. To determine if an attack has occurred, you need to have your administrator evaluate the URL to determine if it is a legitimate download URL.”

Let’s continue with our DPL pattern and extract the full URL where the payload was downloaded and the relevant filenames. Again, we’re opening up the DPL Architect and creating the following pattern:

'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file

With the created pattern, we’re extracting the url and the tarball’s name as download_file into separate fields using the look-around functionality of DPL. Additionally, we’re extracting the filename from the destination file path as well as the whole saved_file value for simpler analysis.

DPL pattern extracts the following result set from the log record:

Applying this pattern and filtering the records as described in the blog post gives you the following DQL query:

fetch logs
| filter log.source == "/var/log/loginsight/runtime.log"
| parse content, """'[' timestamp('yyyy-MM-dd HH:mm:ss.SZ'):ts '] [' ld '] [' ld:class '] [' ld:action ']'"""
| filter class == "com.vmware.loginsight.daemon.commands.SystemCommands"
| parse action, """'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file"""
| filter download_file == filename
| fields ts, url, saved_file

The query produces the following result, which you can now pass on to your administrator for manual evaluation.

A longer description of how the VMware Aria Operations for Logs exploitation works can be found in the Horizon3 blog post.

Further reading about how to use DPL Architect for security use-cases can be found at Speed up your security investigations with DPL Architect.

How to mitigate the Aria Operations for Logs vulnerability exploit

To mitigate this exploit, update to the latest version of VMware Aria Operations for Logs by applying the latest patches. Or follow the instructions within the VMSA-2023-0021.

Sources

1. VMware vRealize Log Insight VMSA-2023-0001 IOCs
2. GitHub – horizon3ai/CVE-2023-34051: VMware Aria Operations for Logs CVE-2023-34051
3. VMware Aria Operations for Logs CVE-2023-34051 Technical Deep Dive and IOCs
4. VMSA-2023-0021
5. Alert: PoC Exploits Released for Citrix and VMware Vulnerabilities

The post Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL appeared first on Dynatrace news.

To help you raise the quality of your investigation results, Dynatrace offers an easy way of structuring data using DPL Architect. This tool lets you quickly extract typed fields from unstructured text (such as log entries) using the Dynatrace Pattern Language (DPL), enabling you to extract timestamps, determine status codes, identify IP addresses, or work with real JSON objects. This allows you to answer even the most complex questions with ultimate precision. The best thing: the whole process is performed on read when the query is executed, which means you have full flexibility and don’t need to define a structure when ingesting data.

>> Scroll down to see Dynatrace DPL Architect in action (24-second video)

Investigating log data with the help of DQL

Let’s look at a practical example. The simplest log analysis use cases can be solved in seconds by applying a simple filter to the query results. If a CISO asks, “Have we seen the IP address 40.30.20.1 in our logs within the last year?” a simple DQL query seems to suffice:

fetch logs, from: -365d
| filter contains(content, “40.30.20.1”)

This query returns all the records that contain the string value 40.30.20.1.

In the next step, adding DQL aggregation functions enables you to answer more complex questions like “How many times in an hour has this IP address visited our website within the last two weeks?” Again, a simple DQL query helps you out:

fetch logs, from: -14d
| filter contains(content, “40.30.20.1”)
| summarize by: bin(timestamp, 1h), count()

However, this kind of simple filtering using a sub-string search is prone to errors and is not precise enough. The mentioned filters return all records that contain 40.30.20.1, but are not necessarily from the source IP portion of the log format. The string 40.30.20.1 might also appear elsewhere, for example, in query parameters, and a simple string filter will return all records containing this search term:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395
40.30.20.109 - - [28/Aug/2023:10:22:04 -0300] "GET / HTTP/1.1" 200 2216

The first log record is matched because of its query parameter—something we don’t care about in our case. The second log record came up because the source IP contains the IP address—this is what we’re really looking for. In a nutshell, if you’re querying terabytes of logs and wish to drill down to specific records, filtering strings from plaintext log content is not enough.

The issue is that questions from CISOs aren’t usually so trivial when it comes to security use cases. Or, the log format where the answers should be looked for is more complex (for example, AWS CloudTrail logs). As a consequence, we need to search structured data to avoid mistakes. To minimize false positives, increase the precision of queries, and get the maximum out of DQL, fields can be extracted from record content.

Extracting patterns using DPL

Dynatrace Pattern Language (DPL) is a pattern language that allows you to describe a schema using matchers, where a matcher is a mini-pattern that matches a certain type of data. These matchers can be used to extract new fields from the schemaless data in Grail to enable more precise log filtering. Consider the same Apache access log example from above:

19.31.99.1 - - [28/Aug/2023:10:27:10 -0300] "GET /index.php?ip=40.30.20.1 HTTP/1.1" 200 3395

To extract the client IP addresses from the beginning of the log, a simple DPL pattern can be used:

IPADDR:client_ip

The IPADDR matches both versions of IP addresses (IPv4 addresses in dot-decimal notation and IPv6 addresses in hextet notation), leaving all other IP-like strings unmatched. For example, if the log record contains a dot-decimal number that is NOT an IPv4 address (999.999.999.999), it won’t be matched.

DPL patterns can be applied in DQL using the parse command. The simplest way to get started with DPL is to use Dynatrace DPL Architect.

DPL Architect to the rescue

DPL Architect is a handy tool, accessible through the Notebooks app, which supports you in quickly extracting fields from records. It helps create patterns, provides instant feedback, and allows you to save and reuse DPL patterns, for faster access to data analytics use cases.

To open the DQL Architect, you have to execute a DQL query, select the content, and choose Extract fields.

Figure 1: Extract fields in Notebooks using DPL Architect (24-second video)

Starting with preset patterns

The simplest way to extract data is using one of the ready-to-use preset patterns available for the most popular technologies, such as AWS, Microsoft, or GCP. Start exploring AWS VPC Flow logs or analyzing Kubernetes audit logs by choosing the patterns from the panel on the left side. You can also customize the list by adding your own individual patterns.

Developing a new pattern using DPL Architect

DPL Architect helps you by creating your own patterns and provides instant feedback. Start typing a DPL expression in the pattern field, and review the matching data (highlighted below) in the preview editor.

For accessing the extracted fields in the resultset, it’s necessary to add a name. In the example below, we’re only interested in the client_ip and the related response_code. The pattern matches the whole record, but only two fields are being extracted since they have defined extract names. Any other data between those two fields won’t be visible in the results (however, it is very easy to add them later if necessary).

Remember, we started our journey with the DPL Architect by selecting query results in a notebook. The colored bar at the top of the DPL Architect shows how many records from your query result match the current DPL pattern. This enables you to modify the DPL pattern to ensure it matches all records in the resultset.

Records that don’t match the current pattern can easily be added to the Match preview panel by selecting Add to preview.

After changing the DPL pattern, any previously unmatched records will be highlighted and the progress bar on the top informs you that 100% of the records in the Notebooks resultset are matched. It’s now time to insert our pattern into the DQL query by selecting Insert Pattern.

Precise extraction of fields from complex data

DPL also provides matchers for more complex data structures, like key-value pairs, structures, and JSON objects, that enable you to parse individual sub-elements from the whole object. Consider the following log record:

1693230219 230.4.130.168 C "GET / HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" 186 {"username":"james","result":0 }

Imagine your CISO comes with a request: “Give me the list of all IP addresses where the user ‘james’ has successfully logged in from.” Extracting these three fields and building a DQL query is as easy as pie:

LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)

The DPL will extract three fields: an IP address from the record, a string from the JSON object element username, and an integer from the JSON element result.

Summarizing the results based on these fields can be done with the following DQL:

fetch logs
| parse content, "LD IPADDR:src_ip LD JSON{STRING+:username, INT:result}(flat=true)"
| filter username == "james" and result == 0
| summarize by: src_ip, count()

Imagine that you don’t have DPL available and need to filter all log records based on only searching for string values: the result would contain a lot of false positives (which only add “noise” to time-critical investigations). This is especially true with more complex log records containing nested JSON objects. The above example was oversimplified intentionally, but considering complex log records like AWS CloudTrail log records, having precise access to data in a specific node of an object makes a huge difference.

Summary

When performing security investigations or threat-hunting activities, it’s important to have precision in place to get reliable results. Historical data needs to be available, and access to object details is required for precise answers. DPL Architect enables you to quickly create DPL patterns, speeding up the investigation flow and delivering faster results. With the possibility of using Dynatrace-provided patterns for selected technology stacks, investigators can deliver answers even faster!

Check out the following video, where Andreas Grabner and I teamed up for a new episode of Dynatrace Observability Clinic. In this video, we dig deeper into the topic of extracting data via DPL, including a live demonstration of DPL Architect.

Referral links

• Documentation: Dynatrace Pattern Language
• Documentation: Dynatrace Pattern Language Architect
• Blog post: Dynatrace Grail: The data lakehouse for observability, security analysis, and automation
• Blog post: Log forensics: Finding malicious activity in multicloud environments with Dynatrace Grail

The post Speed up your security investigations with DPL Architect appeared first on Dynatrace news.