Skip to technology filters Skip to main content
Dynatrace Hub

Extend the platform,
empower your team.

Popular searches:
Home hero bg
HadoopHadoop
Hadoop

Hadoop

Enhanced insights for Hadoop HDFS and Yarn services.

Extension
Free trialDocumentation
Overview DashboardExtension configuration and featuresetsExtra Hadoop metricsHadoop HDFS service entityHadoop Yarn service entityProcess Group Injection
  • Product information
  • Release notes

Overview

Hadoop monitoring in Dynatrace provides a high-level overview of the main Hadoop components within your cluster. These enhanced insights provides additional metrics directly from the HDFS and YARN services.

Use cases

The extension enables insights into the overall health of Hadoop HDFS and YARN services

Get started

For more information on the installation and configuration, please see Hadoop extension in the Dynatrace Documentation.

Details

Compatibility information

  • Hadoop version 2.4.1+
  • Linux OS
  • For full Hadoop visibility, OneAgent must be installed on all machines running the following Hadoop processes: NameNode, ResourceManager, NodeManager, DataNode, and MRAppMaster
Dynatrace
Documentation
By Dynatrace
Dynatrace support center
Subscribe to new releases
Copy to clipboard

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included
Metric nameMetric keyDescriptionUnit
AllocatedContainershadoop.yarn.resourcemanager.AllocatedContainersQueue Current number of allocated containersCount
AllocatedMBhadoop.yarn.resourcemanager.AllocatedMBCurrent allocated memory in MBMegaByte
AllocatedVCoreshadoop.yarn.resourcemanager.AllocatedVCoresCurrent allocated CPU in virtual coresCount
AppsCompletedhadoop.yarn.resourcemanager.AppsCompleted.countTotal number of completed applicationsCount
AppsFailedhadoop.yarn.resourcemanager.AppsFailed.countTotal number of failed applicationsCount
AppsKilledhadoop.yarn.resourcemanager.AppsKilled.countTotal number of killed applicationsCount
AppsPendinghadoop.yarn.resourcemanager.AppsPending.countCurrent number of applications that have not yet been assigned by any containersCount
AppsRunninghadoop.yarn.resourcemanager.AppsRunning.countCurrent number of running applicationsCount
AppsSubmittedhadoop.yarn.resourcemanager.AppsSubmitted.countTotal number of submitted applicationsCount
AvailableMBhadoop.yarn.resourcemanager.AvailableMBCurrent available memory in MBCount
AvailableVCoreshadoop.yarn.resourcemanager.AvailableVCoresCurrent available CPU in virtual coresCount
PendingMBhadoop.yarn.resourcemanager.PendingMBCurrent memory requests in MB that are pending to be fulfilled by the schedulerMegaByte
PendingVCoreshadoop.yarn.resourcemanager.PendingVCoresCurrent CPU requests in virtual cores that are pending to be fulfilled by the schedulerCount
ReservedMBhadoop.yarn.resourcemanager.ReservedMBCurrent reserved memory in MBMegaByte
ReservedVCoreshadoop.yarn.resourcemanager.ReservedVCoresCurrent reserved CPU in virtual coresCount
Metric nameMetric keyDescriptionUnit
NumberOfMissingBlockshadoop.hdfs.namenode.NumberOfMissingBlocks-Count
CacheCapacityhadoop.hdfs.namenode.CacheCapacityThe total cache capacity of all DataNodesByte
CacheUsedhadoop.hdfs.namenode.CacheUsedThe total cache used by all DataNodesByte
Metric nameMetric keyDescriptionUnit
NumActiveNMshadoop.yarn.resourcemanager.NumActiveNMsCurrent number of active NodeManagersCount
NumDecommissioningNMshadoop.yarn.resourcemanager.NumDecommissionedNMsCurrent number of NodeManagers being decommissionedCount
NumLostNMshadoop.yarn.resourcemanager.NumLostNMsCurrent number of lost NodeManagers for not sending heartbeatsCount
NumRebootedNMshadoop.yarn.resourcemanager.NumRebootedNMsCurrent number of rebooted NodeManagersCount
NumUnhealthyNMshadoop.yarn.resourcemanager.NumUnhealthyNMsCurrent number of unhealthy NodeManagersCount
Metric nameMetric keyDescriptionUnit
FilesTotalhadoop.hdfs.namenode.FilesTotalCurrent number of files and directoriesCount
PendingReplicationBlockshadoop.hdfs.namenode.PendingReplicationBlocksCurrent number of blocks pending to be replicatedCount
UnderReplicatedBlockshadoop.hdfs.namenode.UnderReplicatedBlocksCurrent number of blocks under replicatedCount
ScheduledReplicationBlockshadoop.hdfs.namenode.ScheduledReplicationBlocksCurrent number of blocks scheduled for replicationsCount
NumLiveDataNodeshadoop.hdfs.namenode.NumLiveDataNodesNumber of datanodes which are currently liveCount
NumDeadDataNodeshadoop.hdfs.namenode.NumDeadDataNodesNumber of datanodes which are currently deadCount
NumDecomLiveDataNodeshadoop.hdfs.namenode.NumDecomLiveDataNodesNumber of datanodes which have been decommissioned and are now liveCount
NumDecomDeadDataNodeshadoop.hdfs.namenode.NumDecomDeadDataNodesNumber of datanodes which have been decommissioned and are now deadCount
VolumeFailuresTotalhadoop.hdfs.namenode.VolumeFailuresTotalTotal number of volume failures across all DatanodesCount
EstimatedCapacityLostTotalhadoop.hdfs.namenode.EstimatedCapacityLostTotalAn estimate of the total capacity lost due to volume failuresCount
NumDecommissioningDataNodeshadoop.hdfs.namenode.NumDecommissioningDataNodesNumber of datanodes in decommissioning stateCount
NumStaleDataNodeshadoop.hdfs.namenode.NumStaleDataNodesNumber of datanodes marked stale due to delayed hearbeat.Count
Metric nameMetric keyDescriptionUnit
JobsCompletedhadoop.yarn.mrappmaster.JobsCompletedNumber of completed jobsCount
JobsFailedhadoop.yarn.mrappmaster.JobsFailedNumber of failed jobsCount
JobsKilledhadoop.yarn.mrappmaster.JobsKilledNumber of killed jobsCount
JobsPreparinghadoop.yarn.mrappmaster.JobsPreparingNumber of preparing jobsCount
JobsRunninghadoop.yarn.mrappmaster.JobsRunningNumber of running jobsCount
MapsCompletedhadoop.yarn.mrappmaster.MapsCompletedNumber of maps completedCount
MapsFailedhadoop.yarn.mrappmaster.MapsFailedNumber of maps failedCount
MapsKilledhadoop.yarn.mrappmaster.MapsKilledNumber of maps killedCount
MapsRunninghadoop.yarn.mrappmaster.MapsRunningNumber of maps runningCount
MapsWaitinghadoop.yarn.mrappmaster.MapsWaitingNumber of maps waitingCount
ReducesCompletedhadoop.yarn.mrappmaster.ReducesCompletedNumber of completed reducesCount
ReducesFailedhadoop.yarn.mrappmaster.ReducesFailedNumber of failed reducesCount
ReducesKilledhadoop.yarn.mrappmaster.ReducesKilledNumber of killed reducesCount
ReducesRunninghadoop.yarn.mrappmaster.ReducesRunningNumber of running reducesCount
ReducesWaitinghadoop.yarn.mrappmaster.ReducesWaitingNumber of waiting reducesCount
Metric nameMetric keyDescriptionUnit
CapacityTotalhadoop.hdfs.namenode.CapacityTotalCurrent raw capacity of DataNodes in bytesByte
CapacityUsedhadoop.hdfs.namenode.CapacityUsedCurrent used capacity across all DataNodes in bytesByte
CapacityRemaininghadoop.hdfs.namenode.CapacityRemainingCurrent remaining capacity in bytesByte
TotalLoadhadoop.hdfs.namenode.TotalLoadCurrent number of connectionsCount
BlocksTotalhadoop.hdfs.namenode.BlocksTotalCurrent number of allocated blocks in the systemCount
PendingDeletionBlockshadoop.hdfs.namenode.PendingDeletionBlocksCurrent number of blocks pending deletionCount
CorruptBlockshadoop.hdfs.namenode.CorruptBlocksCurrent number of blocks with corrupt replicas.Count
CapacityUsedNonDFShadoop.hdfs.namenode.CapacityUsedNonDFSCurrent space used by DataNodes for non DFS purposes in bytesCount
Metric nameMetric keyDescriptionUnit
ShuffleConnectionshadoop.yarn.nodemanager.ShuffleConnections.countNumber of Shuffle connectionsCount
ShuffleOutputByteshadoop.yarn.nodemanager.ShuffleOutputBytes.countTotal shuffle output bytesByte
ShuffleOutputsFailedhadoop.yarn.nodemanager.ShuffleOutputsFailed.countNumber of failed shuffle outputsCount
ShuffleOutputsOKhadoop.yarn.nodemanager.ShuffleOutputsOK.countNumber of OK shuffle outputsCount
Metric nameMetric keyDescriptionUnit
DataNode BlocksReadhadoop.hdfs.datanode.BlocksRead.countTotal number of blocks read from DataNodeCount
DataNode BlocksRemovedhadoop.hdfs.datanode.BlocksRemoved.countTotal number of blocks removed from DataNodeCount
DataNode BlocksReplicatedhadoop.hdfs.datanode.BlocksReplicated.countTotal number of blocks replicatedCount
DataNode BlocksVerifiedhadoop.hdfs.datanode.BlocksVerified.countTotal number of blocks verifiedCount
DataNode BlocksWrittenhadoop.hdfs.datanode.BlocksWritten.countTotal number of blocks written to DataNodeCount
DataNode BytesReadhadoop.hdfs.datanode.BytesRead.countTotal number of bytes read from DataNodeByte
DataNode BytesWrittenhadoop.hdfs.datanode.BytesWritten.countTotal number of bytes written to DataNodeByte
Metric nameMetric keyDescriptionUnit
DataNode CacheCapacityhadoop.hdfs.datanode.CacheCapacityThe cache capacity of the DataNodeByte
DataNode CacheUsedhadoop.hdfs.datanode.CacheUsedThe cache used the DataNodeByte
DataNode Capacityhadoop.hdfs.datanode.CapacityCurrent raw capacity of the DataNodes in bytesByte
DataNode DfsUsedhadoop.hdfs.datanode.DfsUsedThe storage space that has been used up by HDFS.Byte
DataNode NumBlocksCachedhadoop.hdfs.datanode.NumBlocksCachedThe number of blocks cached on the DataNodeCount
DataNode NumBlocksFailedToCachehadoop.hdfs.datanode.NumBlocksFailedToCacheThe number of blocks that failed to cache on the DataNodeCount
DataNode NumBlocksFailedToUncachehadoop.hdfs.datanode.NumBlocksFailedToUncacheThe number of failed blocks to remove from cache.Count
DataNode NumFailedVolumeshadoop.hdfs.datanode.NumFailedVolumesNumber of failed volumes.Count
DataNode Remaininghadoop.hdfs.datanode.RemainingThe remaining DataNode disk space left in PercentPercent
Metric nameMetric keyDescriptionUnit
NodeManager AllocatedContainershadoop.yarn.nodemanager.AllocatedContainersNodeManager Current number of allocated containersCount
AllocatedGBhadoop.yarn.nodemanager.AllocatedGBCurrent allocated memory in GBGigaByte
AvailableGBhadoop.yarn.nodemanager.AvailableGBCurrent available memory in GBGigaByte
ContainersCompletedhadoop.yarn.nodemanager.ContainersCompleted.countTotal number of successfully completed containersCount
ContainersFailedhadoop.yarn.nodemanager.ContainersFailed.countTotal number of failed containersCount
ContainersInitinghadoop.yarn.nodemanager.ContainersIniting.countCurrent number of initializing containersCount
ContainersKilledhadoop.yarn.nodemanager.ContainersKilled.countTotal number of killed containersCount
ContainersLaunchedhadoop.yarn.nodemanager.ContainersLaunched.countTotal number of launched containersCount
ContainersRunninghadoop.yarn.nodemanager.ContainersRunning.countCurrent number of running containersCount
Metric nameMetric keyDescriptionUnit
FilesAppendedhadoop.hdfs.namenode.FilesAppended.countTotal number of files appendedCount
FilesCreatedhadoop.hdfs.namenode.FilesCreated.countTotal number of files and directories created by create or mkdir operationsCount
FilesDeletedhadoop.hdfs.namenode.FilesDeleted.countTotal number of files and directories deleted by delete or rename operationsCount
FilesRenamedhadoop.hdfs.namenode.FilesRenamed.countTotal number of rename operations (NOT number of files/dirs renamed)Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.
ReleaseDate

Full version history

  • New Platform/Gen3 dashboard bundled with extension
  • Added DQL support for Platform/Gen3 screens
  • Enriched entities with dt.security_context attribute

Full version history

No release notes

Dynatrace Hub
Hub HomeGet data into DynatraceBuild your own app
Dynatrace Intelligence - Agentic Operations SystemThe Dynatrace Agentic AI ecosystem
All (914)Log Management and AnalyticsKubernetesAI and LLM ObservabilityInfrastructure ObservabilitySoftware DeliveryApplication SecurityBusiness ObservabilityDigital Experience
Filter
Type
Built and maintained by
Deployment model
SaaS
  • SaaS
  • Managed
Partner FinderBecome a partnerDynatrace Developer

Application Observability

Optimize your service performance, innovate faster, and collaborate efficiently with automatic and intelligent observability at scale and end-to-end distributed tracing for cloud-native workloads and enterprise apps.

Essentials

Get started with these popular tools for monitoring your applications and microservices.

OpenTelemetry logo

OpenTelemetry

Ingest and analyze OTel traces & metrics, set health alerts & view in context.

Serverless logo

Serverless

Seamless integration with serverless technologies to eliminate blind spots through OOTB integrations with AWS, Azure, and Google Cloud.

Kubernetes logo

Kubernetes

All-in-one Kubernetes observability for K8s infrastructure and apps teams.

Distributed Tracing logo

Distributed Tracing

Analyze and slice distributed traces by any attribute and from any source.

Runtimes

Java logo

Java

Optimize Java performance with E2E monitoring and AI-driven root cause analysis.

.NET logo

.NET

Automatic end-to-end observability for .NET applications and processes.

Go logo

Go

Automatic and intelligent end-to-end observability for Go applications.

Node.js logo

Node.js

Automatic and intelligent end-to-end observability for Node.js applications.

Python logo

Python

Trace your Python apps and services with real-time metrics and automatic RCA.

PHP logo

PHP

Automatic and intelligent end-to-end observability for PHP applications.

Observability frameworks

OpenTelemetry logo

OpenTelemetry

Ingest and analyze OTel traces & metrics, set health alerts & view in context.

Micrometer logo

Micrometer

Ingest Spring metrics and put them in context of trace, log and diagnostics data.

Prometheus logo

Prometheus

Unlock Prometheus power with intelligent analytics and automated insights.

Fluentd logo

Fluentd

Stream log data to Dynatrace via Fluentd for analysis.

Fluent Bit logo

Fluent Bit

Stream logs to Dynatrace via Fluent Bit for analysis and AI observability.

Logstash logo

Logstash

Stream log data from Logstash and analyze it in Dynatrace.

Serverless

AWS Lambda logo

AWS Lambda

Automatic and intelligent end-to-end observability of AWS Lambda traces and logs.

Azure Functions logo

Azure Functions

Observability for serverless and hybrid environments using Azure Functions.

Google Cloud Functions logo

Google Cloud Functions

E2E observability for serverless and hybrid environments using Google Functions.

Azure App Service logo

Azure App Service

Cloud platform for hosting websites, created and operated by Microsoft.

Azure Container Instance logo

Azure Container Instance

Deploy and manage serverless containers on the Microsoft Azure cloud, without having to manage any underlying infrastructure.

Google Cloud Run logo

Google Cloud Run

Dynatrace provides AI-powered observability into serverless containerized apps running on Google Cloud Run.

Databases

Redis logo

Redis

Automatically and intelligently observe, analyze & optimize your Redis server.

MySQL logo

MySQL

Automatically and intelligently observe, analyze and optimize how your the usage, health and performance of your database.

Apache Cassandra logo

Apache Cassandra

Improve Apache Cassandra observability.

Couchbase logo

Couchbase

Couchbase Capella (SaaS) database monitoring.

Message queues

Apache Kafka logo

Apache Kafka

Automatic and intelligent observability with trace and metric insights.

RabbitMQ logo

RabbitMQ

Automatic end-to-end traces of producers and consumers for RabbitMQ.

Apache ActiveMQ Classic logo

Apache ActiveMQ Classic

Automatic and smart observability for ActiveMQ with trace and metric insights.

Apache ActiveMQ Artemis logo

Apache ActiveMQ Artemis

ActiveMQ Artemis observability with end-to-end producers and consumers tracers.

Microsoft Message Queuing (MSMQ) logo

Microsoft Message Queuing (MSMQ)

Observability for MSMQ with end-to-end traces of connected producers/consumers.

Application stacks

Reactor Core logo

Reactor Core

End-to-end trace your applications when the Reactor Core publishers Mono and Flux are used.

Spring logo

Spring

End-to-end observability for Java Spring apps, automatic and intelligent.

Netty logo

Netty

End-to-end observability for your Netty web framework with code-level insight.

IBM WebSphere Application Server logo

IBM WebSphere Application Server

Automatically and intelligently monitor, analyze, and optimize your application server and all applications deployed anywhere in your stack.

More resources

Adobe Experience Manager Cloud Service logo

Adobe Experience Manager Cloud Service

End-to-end observability for your Adobe Experience Manager Cloud Service.

Adobe PhoneGap logo

Adobe PhoneGap

Monitor hybrid mobile apps built with Adobe PhoneGap running on iOS or Android.

Amazon Corretto logo

Amazon Corretto

Automatically and intelligently monitor, analyze, and optimize the performance of your virtual machine.

AngularJS logo

AngularJS

JavaScript-based open-source front-end web framework mainly maintained by Google.

Apache Axis2 logo

Apache Axis2

Core engine for Web services.

Apache Camel logo

Apache Camel

Intelligently monitor, analyze, and optimize your integration framework and all applications deployed in your stack.

Apache CXF logo

Apache CXF

Open-source, fully featured Web services framework.

Apache OpenEJB logo

Apache OpenEJB

Automatically and intelligently monitor, analyze and optimize your applications based on Apache OpenEJB.

Apache Tomcat logo

Apache Tomcat

Automatically and intelligently monitor, analyze, and optimize your application server and all applications deployed anywhere in your stack.

Apache TomEE logo

Apache TomEE

Automatically and intelligently monitor, analyze, and optimize your application server and all applications deployed anywhere in your stack.

Apache Zookeeper logo

Apache Zookeeper

Monitor and analyze your Apache Zookeeper with this JMX-based extension.

Apigee logo

Apigee

Remotely monitor your Apigee proxies.

Azul Platform Core (Zulu) logo

Azul Platform Core (Zulu)

Automatically and intelligently monitor, analyze, and optimize the performance of your virtual machine.

Azul Platform Prime (Zing) logo

Azul Platform Prime (Zing)

Automatically and intelligently monitor, analyze, and optimize the performance of your virtual machine.

Backbone.js logo

Backbone.js

JavaScript library based on the Model–view–presenter application design paradigm.

BellSoft Liberica logo

BellSoft Liberica

Automatically and intelligently monitor, analyze, and optimize the performance of your virtual machine.

BOSH bpm logo

BOSH bpm

Automated monitoring of platform processes running in BOSH bpm containers.

C logo

C

Procedural programming language supporting lexical variable scope, and recursion.

CakePHP logo

CakePHP

Automatically and intelligently monitor, analyze, and optimize your applications developed with CakePHP.

CentOS logo

CentOS

Community-supported computing platform compatible with Red Hat Enterprise Linux.

IBM CICS Transaction Gateway for z/OS logo

IBM CICS Transaction Gateway for z/OS

Intelligently monitor your transactions end-to-end and analyze the performance of your CICS Transaction Gateway on z/OS.

Cloud Foundry logo

Cloud Foundry

Harness automation and AI to simplify observability on Cloud Foundry at scale.

Connection Pools: JBoss logo

Connection Pools: JBoss

Application server method of pooling and sharing connections to a database.

Connection Pools: Apache Commons logo

Connection Pools: Apache Commons

Application method of pooling and sharing multiple connections to a database.

Connection Pools: C3P0 logo

Connection Pools: C3P0

Application server method of pooling and sharing connections to a database.

Tomcat (JMX) logo

Tomcat (JMX)

JMX monitoring of Tomcat connection and thread pools, and web request activity.

Connection Pools: WebLogic logo

Connection Pools: WebLogic

Application server method of pooling and sharing connections to a database.

Connection Pools: WebSphere Liberty logo

Connection Pools: WebSphere Liberty

Application server method of pooling and sharing connections to a database.

Consul Service Mesh (StatsD) logo

Consul Service Mesh (StatsD)

Extend visibility into your Consul Service Mesh instances to monitor health and improve performance.

containerd logo

containerd

Distributed tracing and metrics for services in containerd in Kubernetes.

Red Hat Enterprise Linux CoreOS logo

Red Hat Enterprise Linux CoreOS

Open-source lightweight OS based on the Linux, providing infrastructure to clustered deployments.

cri-o logo

cri-o

Distributed tracing and metrics for services in cri-o containers in Kubernetes.

Observability for Developers on Cursor logo

Observability for Developers on Cursor

Get Real time Code-Level data directly to your Cursor IDE.

Debian logo

Debian

Association of individuals who have made common cause to create a free operating system.

Disk Analytics logo

Disk Analytics

This extension allows more detailed visibility on Linux hosts' local datastores where OneAgent is installed.

Drupal logo

Drupal

Web content management for back-end framework.

Dynatrace Bindplane logo

Dynatrace Bindplane

Optimize and manage your telemetry at the edge before it reaches Dynatrace.

Eclipse Jetty logo

Eclipse Jetty

Automatically and intelligently monitor and analyze your Jetty applications.

Eclipse OpenJ9 logo

Eclipse OpenJ9

Automatically and intelligently monitor, analyze, and optimize the performance of your virtual machine.

Eclipse Temurin (Adoptium) logo

Eclipse Temurin (Adoptium)

Automatically and intelligently monitor, analyze, and optimize the performance of your virtual machine.

Ember.js logo

Ember.js

Open-source JavaScript web framework, based on the Model–view–viewmodel pattern.

etcd for OpenShift logo

etcd for OpenShift

Get deep insights into your OpenShift control plane using etcd metrics.

Express logo

Express

Fast, unopinionated, minimalist web framework for Node.js.

Fedora logo

Fedora

Linux distribution sponsored primarily by Red Hat, a subsidiary of IBM.

Garden-RunC logo

Garden-RunC

Automated distributed tracing and metrics for microservices running in garden runc containers in Cloud Foundry.

GlassFish logo

GlassFish

Automatically and intelligently monitor, analyze, and optimize your application server and all applications deployed anywhere in your stack.

GraalVM Native Image logo

GraalVM Native Image

Intelligently monitor and optimize the performance of your native Java apps.

gRPC logo

gRPC

Open source remote procedure call (RPC) system.

Are you looking for something different?

We have hundreds of apps, extensions, and other technologies to customize your environment

Extend your knowledge

Discover distributed tracing

Discover distributed tracing

Observe requests across distributed systems and microservices and generate high-quality data for analysis.
Read more
Discover OpenTelemetry

Discover OpenTelemetry

Ingest and chart custom OpenTelemetry traces and metrics, set alerts, and analyze them with traces, logs, and diagnostics data.
Read more
Profile and optimize

Profile and optimize

Go beyond automated problem detection to manually detect problems with the analysis toolset for profiling and optimization.
Read more