Remote JMX Monitoring

The Remote JMX Monitoring agent connects to a configured JMX server. Remote JMX Monitoring reports JMX metrics and includes these additional capabilities:
  • Monitors multiple remote JMX servers
  • Provides a secure JMX connection
  • Provides configurable JMX metric names and values for specific managed beans (MBeans)
This article contains the following topics:
2
Download and Extract the Infrastructure Agent
After you get access to
DX APM
, you download and extract Remote JMX Monitoring.
Follow these steps:
  1. Log in to
    DX SaaS
    .
  2. Select
    Open
    on the
    Application Performance Management
    box. The Application Performance Management home page appears.
  3. Download the Infrastructure Agent package that contains Remote JMX Monitoring.
    1. At the top center of the Application Performance Management page, select
      Download Agent
      .
    2. On the
      Agent Selection
      page, select the tab corresponding to your operating system.
    3. Select
      Infrastructure
      for the environment.
    4. Select
      Remote JMX Monitoring
      as the agent.
    5. (Optional) Configure the appropriate property now. You can also configure the property later in the Infrastructure Agent
      bundle.properties
      file.
      introscope.agent.remotejmx.system.s1.hostport
      Remote JMX system host:port values for the Remote JMX agent to monitor the available JMX servers. Separate multiple values using commas.
      introscope.agent.remotejmx.server.s1.name
      System name of the Remote JMX server that the Remote JMX agent monitors. Separate multiple values using commas.
    6. (Optional) To correlate Remote JMX Monitoring with Host monitoring, in the
      Agent Components List
      , select
      Host Monitor
      . Host monitoring is included in the Remote JMX Monitoring package download.
  4. Select
    Download and Install
    .
    DX APM
    downloads the package to your computer.
  5. Verify that the Infrastructure Agent download file contains the Remote JMX Monitoring file.
    UNIX
    :
    <remote_JMX-apmia-set_of_numbers
    _version>.tar
    .
    For example,
    Remote_JMX-apmia-20190708_v1.tar
    .
    Windows
    :
    <remote_JMX-apmia-set_of_numbers>.zip
    For example,
    Remote_JMX-apmia-20190708_v1.zip
    .
    If you cannot find the file, repeat steps 4 and 5.
  6. Extract the downloaded Infrastructure Agent file.
    UNIX
    : Run this command:
    tar -xf <remote_JMX-apmia-set_of_numbers_version>.tar
    .
    For example,
    tar -xf Remote_JMX-apmia-20190708_v1.tar
    .
    Windows
    : Unzip the file.
Install and Run the Infrastructure Agent
You install the Infrastructure Agent before you install Remote JMX Monitoring.
Follow these steps:
  1. Navigate to the
    apmia/bin
    directory.
  2. Run the appropriate command to install the Infrastructure Agent:
    UNIX
    :
    ./apmia-ca-installer.sh install
    Windows
    :
    apmia-ca-installer.bat install
Log Files
You can review the Infrastructure Agent and Remote Monitoring logs in the
<Infrastructure_Agent_Home>/logs
directory.
These log files are in the logs directory:
  • IntroscopeAgent.log
  • APMIAgentConsole.log
Configure Remote JMX Monitoring System Keys and System Name
You configure Remote JMX system keys and system name in the
bundle.properties
file.
  1. Navigate to the
    apmia/extensions/RemoteJmxMonitorExtension
    directory.
  2. Open the
    bundle.properties
    file in a text editor.
  3. Configure remote JMX system keys by setting a comma-separated list in the
    introscope.agent.remotejmx.system.keys
    property.
    1. Configure one key per JMX system type. For example, if you want to monitor two instances of Tomcat and another two instances of JBoss, you set two keys. One key each for Tomcat and JBoss. You configure the property as shown in this example:
      introscope.agent.remotejmx.system.keys=s1,s2
    2. Add the connection details for each system using
      one
      of these two options.
      • Set the host and JMX port for each remote system. In this example, the ports for host1 and host2 are for the Tomcat servers. The ports for host3 and host 4 are for the JBoss servers.
        introscope.agent.remotejmx.system.s1.hostport=host1:1099,host2:1099 introscope.agent.remotejmx.system.s2.hostport=host3:9999,host4:9999
      • Set the JMX URL for the JMX server that is registered to a Java RMI server with a custom service name. The default JMX server service name value is
        jmxrmi
        . For example:
        introscope.agent.remotejmx.system.s1.jmxurl=service:jmx:rmi:///jndi/rmi://host1:1099/foo
  4. Configure the remote JMX system name. Set the name of the remote JMX server, which displays in the Map and Metric Tree. The default value is
    JMX System
    , as shown in this example:
    introscope.agent.remotejmx.system.<s1>.name=JMX System
  5. Save and close the file.
  6. Restart the Infrastructure Agent using the appropriate command:
    UNIX
    :
    ./apmia-ca-installer.sh restart
    Windows
    :
    apmia-ca-installer.bat restart
  7. (Optional) Use the Remote Monitoring the
    jmx port
    and
    jmx system name
    attributes to create Remote Monitoring perspectives.
Configure Remote JMX Monitoring Security
You can configure Remote JMX Monitoring to pass encrypted credentials to connect with the remote JMX server.
Follow these steps:
  1. Navigate to the
    apmia/extensions/RemoteJmxMonitorExtension
    directory.
  2. Open the
    bundle.properties
    file in a text editor.
  3. Set the security-related properties depending on the organization or environment requirements.
    • introscope.agent.remotejmx.ssl.keyStore
      The keystore file name. The KeyStore file is required when both SSL is enabled and the JMX server has client authentication that is enabled. Ensure that the file is located in the
      <ext-home>/config/security
      directory.
      Default:
      keystore
    • introscope.agent.remotejmx.ssl.keyStorePassword
      The password to open the KeyStore file.
      Default:
      None
    • introscope.agent.remotejmx.ssl.trustStore
      The trustStore file name. The trustStore file is required when SSL is enabled. Ensure that the file is located in the
      <ext-home>/config/security
      directory.
      Default:
      trustStore
    • introscope.agent.remotejmx.ssl.trustStorePassword
      The password to open the trustStore file.
      Default:
      None
    • introscope.agent.remotejmx.system.s1.authentication.password
      Password for file-based JMX authentication.
      Default:
      None
    • introscope.agent.remotejmx.system.s1.authentication.username
      Username for file-based JMX authentication.
      Default:
      None
    • introscope.agent.remotejmx.system.s1.ssl
      Set this property to
      true
      when SSL security is enabled on the remote JMX server.
      Default:
      false
  4. Save and close the file.
  5. Restart the Infrastructure Agent using the appropriate command:
    UNIX
    : ./apmia-ca-installer.sh restart
    Windows
    : apmia-ca-installer.bat restart
Tip:
If you want to pass encrypted credentials for the connection to the remote JMX server, run this command:
java -cp lib/Agent.jar com.wily.util.properties.PropertiesUtils encrypt <clear text key>
Configure Remote JMX Monitoring Metric Reporting
The Remote JMX Monitoring agent reports metrics in three ways:
  • Default reporting The agent reports all JMX MBeans by default. Use the default reporting when you do not have knowledge about the JMX MBeans on your remote servers.
  • The agent reports selective JMX MBeans having default metric names and values For this option, you black list and white-list specific JMX MBeans by configuring properties.
  • The agent reports custom metric names and values. For this option, you create a configuration module containing custom metric names with values for the Remote JMX Monitoring agent to report.
Determine one or more reporting methods that are most useful for your environment.
Remote JMX Monitoring Default Reporting and General Metrics
Use Remote JMX Monitoring default reporting when you do not have knowledge about the JMX MBeans on your remote servers. Consider these factors when evaluating using the default reporting:
  • Metric names might not be intuitive
  • You cannot manipulate the values that the agent has gathered from remote JMX MBeans.
  • The Remote JMX Monitoring agent reports data only of the numeric and string types. The agent ignores data in which the JMX MBeans attribute values are of a custom type.
Remote JMX Monitoring is enabled by default. We provide the default
jvm
directory by default. This directory contains the
jmx.yml
file responsible for monitoring JMX beans.
Here is the metric path for all the managed beans:
<infra-agent>|<metricRootNode>|<domain>|<key-value-pair>:<attribute>
Example
Imagine that the Infrastructure Agent is running on apmia-host. You have a Java application running on host1 configured with JMX port 1999 as shown in this example:
introscope.agent.remotejmx.system.s1.hostport=host1:1099
The Java application MBean object name is
com.example.domain:type=Foo,name=Bar
. The MBean attribute is
size
.
Remote JMX Monitoring reports this MBean path and metric:
apmia-host|Infrastructure|Agent|JMX|JMX System on host1_1999|com.example.domain|name=bar|type=foo:size
Follow these steps:
  1. Navigate to the
    apmia/extensions/RemoteJmxMonitorExtension
    directory.
  2. Open the
    bundle.properties
    file in a text editor.
  3. (Optional) Set how often Remote JMX Monitoring seeks new JMX servers. Configure the
    introscope.agent.remotejmx.server.discovery.interval
    property.
    • introscope.agent.remotejmx.server.discovery.interval
      How often, in milliseconds, that Remote JMX Monitoring seeks new JMX servers.
      Default:
      jvm (enabled)
      Note:
      You must restart the Infrastructure Agent before changes to this property take effect.
      Example:
      introscope.agent.remotejmx.server.discovery.interval=60000
  4. Save and close the file.
  5. Restart the Infrastructure Agent using the appropriate command:
    UNIX
    :
    ./apmia-ca-installer.sh restart
    Windows
    :
    apmia-ca-installer.bat restart
  6. (Optional) Use the Remote Monitoring
    jmx port
    and
    jmx system name
    attributes to create Remote Monitoring perspectives.
Remote JMX Monitoring Selective JMX MBean Reporting
Remote JMX Monitoring can report metrics by evaluating JMX MBeans using blacklist and white list property values. The Remote JMX Monitoring agent queries JMX MBeans that match the white list property values and are not excluded by blacklist property values.
Here is the metric path for all the managed beans:
<infra-agent>|<metricRootNode>|<domain>|<key-value-pair>:<attribute>
You can blacklist and white-list specific JMX MBeans that the Remote JMX Monitoring agent reports.
Follow these steps:
  1. Navigate to the
    apmia/extensions/RemoteJmxMonitorExtension
    directory.
  2. Open the
    bundle.properties
    file in a text editor.
  3. (Optional) Blacklist the reporting of JMX MBean objects as JMX metrics.
    1. Follow the JMX MBean specifications to define the object names. Do not use semi-colons in the object names.
    2. Configure the
      introscope.agent.remotejmx.system.<system-key>.mbeanPatternsBlackList
      property.
      • introscope.agent.remotejmx.system.<system-key>.mbeanPatternsBlackList
        A semicolon-separated list of JMX MBean object names. The Remote JMX Monitoring ignores all matching MBeans and does not report the MBeans as JMX metrics.
        Default:
        none
        Note:
        You must restart the Infrastructure Agent before changes to this property take effect.
        Example:
        introscope.agent.remotejmx.system.s1.mbeanPatternsBlackList=com.example.domain:type=Legacy;com.example.domain:name=Unknown
  4. (Optional) white list the reporting of JMX MBean objects as JMX metrics.
    1. Follow the JMX MBean specifications to define the object names. Do not use semi-colons in the object names.
    2. Configure the
      introscope.agent.remotejmx.system.<system-key>.mbeanPatternsWhiteList
      property.
      • introscope.agent.remotejmx.system.<system-key>.mbeanPatternsWhiteList
        A semicolon-separated list of MBean object names. The Remote Monitoring agent automatically reports JMX metrics for matching MBeans.
        Default:
        *:*
        Note:
        You must restart the Infrastructure Agent before changes to this property take effect.
        Example:
        introscope.agent.remotejmx.system.s1.mbeanPatternsWhiteList=Users:type=Role,*;Catalina:type=Manager,*
  5. Save and close the file.
  6. Restart the Infrastructure Agent using the appropriate command:
    UNIX
    :
    ./apmia-ca-installer.sh restart
    Windows
    :
    apmia-ca-installer.bat restart
Remote JMX Monitoring Custom JMX Metrics Reporting
A configuration module extends Remote JMX Monitoring. Creating a configuration module allows you to provide custom metric names and values for a given remote JMX component. Usually the configuration module name refers to the component that provides the JMX metrics.
This example module configuration shows how to gather metrics for a hypothetical caching system.
version:
"1.0"
configElements:
- objectName:
"com.foo.server:type=Cache"
attribute:
"RequestsIn"
metricName:
"{rootNode}|{type}:Requests In Per Interval"
metricType:
"counter"
- objectName:
"com.foo.server:type=Cache"
attribute:
"name"
metricName:
"{rootNode}|{type}:{name}"
metricType:
"string"
- objectName:
"com.foo.server:type=Cache,name=MemoryBytes"
attribute:
"size"
metricName:
"{rootNode}|{type}:Cache Size (mb)"
metricType:
"counter"
valueFactor:
0.000001
valueAddition:
0
- objectName:
"com.foo.server:type=Cache,name=MemoryBytes"
attribute:
"size"
metricName:
"{rootNode}|{type}:Cache Size Increase Per Interval"
metricType:
"rate"
To create a configuration module, you must determine the metric types, metric names, placeholders. You also configure the
introscope.agent.remotejmx.system.<system-key>.config.modules
property.
Follow these steps:
  1. Navigate to the
    apmia/extensions/RemoteJmxMonitorExtension/config/modules
    directory.
  2. Create a new module directory and name it for the application or component to be monitored. For example,
    cache
    .
    apmia/extensions/RemoteJmxMonitorExtension/config/modules/cache
    Important:
    Every modules directory must contain a
    jmx.yml
    file. The file contents can differ based on the application or component that the Remote JMX agent is monitoring. We recommend that you use the option in Step 3 to create your
    jmx.yml
    file because our code provides some helpful pre-formatted metrics
    .
  3. (Optional) Create a pre-formatted
    jmx.yml
    file.
    1. Using a text editor, create a
      jmx.yml
      file.
    2. Copy the example hypothetical caching system module code block we provide before this procedure.
    3. Add, delete, or change the metrics in the file using instructions from Steps 4b and 4c.
    4. Save and close the file.
  4. (Optional) Create a custom
    jmx.yml
    file.
    1. Use a text editor, create a
      jmx.yml
      file.
    2. Define one or more metric types for your configuration module.
      • Counter Usage:
        Use the counter metric type to report attributes having a numeric return. The value fluctuates and stays at the last known value until new data is available.
        Primitive type values:
        int
        ,
        long
        ,
        float
        ,
        double
        Object types:
        Integer
        ,
        Long
        ,
        Float
        ,
        Double
        Example:
        Imagine that you have a Cache MBean with the
        getSize()
        attribute, which returns the current cache size. The returned value type is
        long
        .
        To report the latest value as a metric, use this configuration:
        - objectName: "com.foo.server:type=Cache,name=MemoryBytes" attribute: "size" metricName: "{rootNode}|{type}:Cache Size (mb)" metricType: "counter"
        This configuration could report a metric value of, for example,
        100 (mb)
        .
      • Rate Usage:
        Use the rate metric type to report an attribute that has an increase from the last known value to the current value. The value fluctuates and stays at the last known value until new data is available.
        Attribute value type:
        numeric
        Returned value type:
        long
        Example:
        Imagine that you have a Cache MBean with the
        getSize()
        attribute, which returns the current cache size. In Interval 1, the cache size is 100 mb. In Interval 2, the cache size increases to 103 mb. To report the value increase as metric, use this configuration:
        - objectName: "com.foo.server:type=Cache,name=MemoryBytes" attribute: "size" metricName: "{rootNode}|{type}:Cache Size (mb) Increase Per Interval" metricType: "rate"
        This configuration reports a metric value of
        3
        .
      • String Usage:
        Use the string type for reporting an attribute having a return type as
        string
        .
        Example:
        Imagine that you have a Cache MBean with the
        getType()
        attribute, which returns the string cache type. To report the latest value, use this configuration:
        - objectName: "com.foo.server:type=Cache,name=MemoryBytes" attribute: "type" metricName: "{rootNode}|{type}:Cache Type" metricType: "string"
        This configuration could report a metric value of, for example, LRU (Least Recently Used).
    3. Use these placeholders to determine one or more custom metric names in your configuration module.
      • attribute
        Name of the attribute from which the agent is gathering data.
      • attributeType
        Type of attribute from which the agent is gathering data.
      • className
        Remote object class name
      • componentName
        Name that is assigned to the remote system in the
        bundle.properties
        file.
      • domain
        MBean object domain name
      • host
        Remote JMX system host name.
      • port
        Remote JMX system JMX port.
      • rootNode
        Root node for a given remote JMX system. The rootNode value is used to display the Metric Tree path.
        For example:
        SuperDomain|<apmia-host>|Infrastructure|Agent|JMX|<jmx-system-name on host_port>
  5. Save and close the
    jmx.yml
    file.
  6. Define the configuration module in
    introscope.agent.remotejmx.system.<system-key>.config.modules
    property .
      • introscope.agent.remotejmx.system.<system-key>.config.modules
        This property provides a comma-separated list of JMX configuration modules.
        Default:
        JVM. By default, the
        JVM
        module is enabled.
        Note:
        You must restart the Infrastructure Agent before changes to this property take effect.
    1. Navigate to the
      apmia/extensions/RemoteJmxMonitorExtension
      directory.
    2. Open the
      bundle.properties
      file in a text editor.
    3. Use
      one
      of these options.
      • (Optional) Add the name of the new directory to the
        introscope.agent.remotejmx.system.s1.config.modules
        property.
        For example, if you created a
        cache
        directory, add
        cache
        as shown in this code sample:
        introscope.agent.remotejmx.system.s1.config.modules=jvm,cache
        In this example, you are keeping the default
        jvm
        directory containing its
        jmx.yml
        file, which monitors JMX beans. You are adding the new
        cache
        directory.
      • (Optional) Add the absolute directory path to the
        introscope.agent.remotejmx.system.s1.config.modules
        property. Here is an example:
        introscope.agent.remotejmx.system.s1.config.modules=/opt/jmx/jvm,/opt/jmx/cache
  7. Save and close the file.
  8. Restart the Infrastructure Agent using the appropriate command:
    UNIX
    :
    ./apmia-ca-installer.sh restart
    Windows
    :
    apmia-ca-installer.bat restart
Monitor Remote JMX Health
You view the Remote JMX instance data using the
Components
view and other views in Application Performance Management.
  1. Log in to
    DX SaaS
    .
  2. Select
    Open
    on the
    Application Performance Management
    box. Application Performance Management home page appears
  3. In the left navigation pane, select the
    Map
    icon.
  4. View the Remote JMX infrastructure layer data.
    1. Select
      Infrastructure Layer
      in upper-left corner drop-down.
    2. Select
      Default
      in the right drop-box
    3. The <
      jmx-system-name>
      node displays in the map. Notice that we group all remote JMX servers that are identified by jmx_system_name. For example if your environment has a Hello World JMS server and a Tomcat JMS server, both servers display on the map within one
      JMV-System-Type
      box.
    4. Select the
      jmx system name
      node to open
      Components
      View
      to the right of the map. Notice that the type is
      JMX_System
      .
    5. Select the
      Jmx System
      tab to view the attributes and metrics from the infrastructure layer. When the Infrastructure Agent is installed with Host Monitoring enabled, the agent uses the remote JMX system host name and IP address attributes to correlate with Host Monitoring.
  5. Scroll down the data to examine the Remote JMX Monitoring metrics to determine if any of the values indicate poor performance.
  6. Examine the Remote JMX Monitoring agent metrics here in the metrics browser:
    SuperDomain|<apmia-host>|Infrastructure|Agent|JMX|<jmx-system-name on host_port>
    Note:
    Remote JMX Monitoring might not report all the metrics that you see with JConsole. This situation occurs because some of the metrics are complex metrics that
    DX APM
    does not support.
Monitoring Zookeeper Cluster With Remote JMX Extension
The Zookeeper service manages multiple nodes in a cluster and when you enable Remote JMX extension on all the nodes, you can monitor and collect metrics for the cluster.
A Zookeeper cluster typically contains the following types of nodes:
  • Leader Node
    - a node that functions as a leader for coordination between all the remaining nodes.
  • Follower Node
    - a node that follows the relationship established by the leader node.
  • Broker Node
    - a node that is connected to the Zookeeper service and relays information between the service and the cluster nodes.
Configure Remote JMX Monitoring For Zookeeper Monitoring
You can configure Remote JMX Monitoring for Zookeeper monitoring in the
bundle.properties
file.
Follow these steps:
  1. Navigate to the
    apmia/extensions/RemoteJmxMonitorExtension
    directory.
  2. Open the
    bundle.properties
    file in a text editor.
  3. Configure remote JMX system keys by setting a comma-separated list in the
    introscope.agent.remotejmx.system.keys
    property.
    For example,
    introscope.agent.remotejmx.system.keys=s1
  4. Configure the connection details for each system in the
    introscope.agent.remotejmx.system.<jmx_system_key>.hostport
    property.
    Set the host and JMX port of the Zookeeper remote system. If it is the Zookeeper cluster, add all nodes host with JMX port.
    For example,
    introscope.agent.remotejmx.system.s1.hostport=zookeeper-host1:9010, zookeeper-host1:9011, zookeeper-host3:9012
  5. Configure the remote JMX system name in the
    introscope.agent.remotejmx.system.<jmx_system_key>.name
    property.
    The name of the remote JMX server displays in the
    Map
    and
    Metric Tree
    views.
    For example,
    introscope.agent.remotejmx.system.s1.name=JMX Remote System
    By default, the value is
    JMX System
    .
  6. Save and close the file.
  7. Restart the Infrastructure Agent using the appropriate command:
    • UNIX:
      ./apmia-ca-installer.sh restart
    • Windows:
      apmia-ca-installer.bat restart
If Zookeeper JMX remote hosts are secured, then follow these steps to Configure Remote JMX Monitoring Security.
Zookeeper Metrics
Metrics Path
:
JMX|<Zookeeper_cluster_name>|Default|<Zookeeper_service_Name>
The following is the list of metrics collected for the Zookeeper cluster and the nodes:
  • Zookeeper Cluster
    • Connection Status
    • StartTime
    • TickTime
    • InitLimit
    • QuorumSize
    • SyncLimit
    • Number of Alive Connections
    • Data Directory Size
  • Leader Node
    • RequestThrottleStallTime
    • RequestThrottleLimit
    • DataDirSize
    • NumAliveConnections
    • TickTime
  • Follower Node
    • DataDirSize
    • NumAliveConnections
    • TickTime
  • Zookeeper Connection/JMX Metrics/Broker Node
    • Nodes
    • SourceIP
    • Avg Latency
    • Packets Sent
    • Packets Received