Implement Enterprise Manager and Cluster Sizing

Contents
apmdevops97
Contents
Identify Enterprise Manager Workload
Use this table to identify the Enterprise Manager workload factors that affect your CA APM environment. The impact increases as the indicated workload factor grows.
Workload Factor
Performance Impact
Metric load
Network bandwidth
Enterprise Manager heap memory
Some agent buffer memory
SmartStor disk space
Harvest duration
SmartStor duration
Metric queries
Network bandwidth
Enterprise Manager heap memory
Client heap memory
Harvest duration
SmartStor duration
Calculators
CPU (depends on size of the metric group).
Enterprise Manager startup time (depends on size of Management Modules).
Client login (depends on size of Management Modules).
Harvest duration
Dashboards
Execute metric queries and calculators (See Metric queries and Calculators).
Application triage map
Small amount of network bandwidth
Small amount of APM database disk storage
Generates metrics (See Metric load)
Execute calculators (See Calculators).
CA CEM data
Network bandwidth
Heap memory on Tim Collection service
CPU on Tim Collection service
Heap memory on Stats Aggregation service
CPU on Stats Aggregation service
APM database disk space
Generates metrics (see Metric load).
Transaction Tracing
Network bandwidth
Enterprise Manager heap memory
Enterprise Manager disk space (traces.db)
Integrations with other CA Technologies products
Enterprise Manager CPU
Executes metric queries (See Metric queries)
Generates metrics (See Metrics load)
You can tune Enterprise Manager and cluster performance in a number of ways. Use this table to determine how to perform various tuning tasks that are based on the performance optimization you require.
Performance Optimization
Tuning Tasks
Optimize SmartStor disk performance
Reduce SmartStor duration
Increase query responsiveness
Set introscope.enterprisemanager.smartstor.directory in IntroscopeEnterpriseManager.properties to a directory on a dedicated disk or dedicated I/O path, and set introscope.enterprisemanager.smartstor.dedicatedcontroller to true.
For high metric load, configure the SmartStor disk to optimize sequential writes.
For best historical query performance, with low to moderate metric load, configure the SmartStor disk to optimize random reads.
Increase Enterprise Manager Java heap size
Edit lax.nl.java.option.additional in Introscope_Enterprise_Manager.lax.
Notes:
Set - Xms and - Xmx to the same value.
The available RAM must be 2 GB larger than the - Xmx value.
For heap sizes larger than 1500 MB on Windows or 2000 MB on Linux, a 64-bit JVM is required.
Configure multiple Enterprise Managers to run on the same computer.
Provide 4 CPU cores for each Enterprise Manager instance.
Provide at least 2 GB RAM for each Enterprise Manager instance.
Provide a dedicated disk I/O path for each Enterprise Manager instance.
Set introscope.enterprisemanager.availableprocessors in IntroscopeEnterpriseManager.properties to the number of CPU cores allocated to each Enterprise Manager instance.
Limit disk space that is required for Transaction Traces.
Reduce value of introscope.enterprisemanager.transactionevents.storage.max.data.age in IntroscopeEnterpriseManager.properties.
Reduce value of introscope.enterprisemanager.transactionevents.storage.max.disk.usage clamp in apm-events-thresholds-config.xml.
Limit Transaction Trace frequency per agent
Reduce value of introscope.enterprisemanager.agent.trace.limit clamp in apm-events-thresholds-config.xml.
Limit disk space that is required for historical metrics.
Adjust introscope.enterprisemanager.smartstor.tier<n> properties in IntroscopeEnterpriseManager.properties.
Adjust application triage map refresh rate.
Adjust introscope.apm.data.agingTime in IntroscopeEnterpriseManager.properties.
Note:
A lower value increases agent and Enterprise Manager resource usage.
Limit disk space that is required for thread dumps.
Adjust introscope.enterprisemanager.threaddump.storage.max.disk.usage in IntroscopeEnterpriseManager.properties.
Limit the number of live metrics an Enterprise Manager processes.
Adjust introscope.enterprisemanager.metrics.live.limit clamp in apm-events-thresholds-config.xml.
Limit the total number of metrics (live and historical) that can be stored on an Enterprise Manager.
Adjust introscope.enterprisemanager.metrics.historical.limit clamp in apm-events-thresholds-config.xml.
Note:
Even though the default value of this property is set to 1.2 million, you can increase the value to 5 million without any significant performance impact.
Limit the number of agent connections that an Enterprise Manager can accept.
Adjust introscope.enterprisemanager.agent.connection.limit clamp in apm-events-thresholds-config.xml.
Limit the number of Command Line Workstation queries that can run concurrently.
Adjust introscope.clw.max.users clamp in apm-events-thresholds-config.xml.
CA APM Deployment Overview
You have many options for deploying Introscope and CA APM.
Since the introduction of the application triage map, monitored application topology has become a significant factor in Enterprise Manager sizing. The topology of monitored applications determines these sizing-related factors:
  • Number of metrics generated
  • Number of calculators that operate on the generated metrics.
  • Application triage map information that CA APM detects and stores in the APM database.
TIMs provide transaction monitoring data, which the Enterprise Manager services process and store in the APM database. Enterprise Manager services are hosted inside the Enterprise Manager. Enterprise Manager services resource requirements are independent of the resource requirements for processing application monitoring data from agents. This situation provides you with a range of CA APM environment configuration options. Depending on your computing environment, the Enterprise Manager services can run in a standalone Enterprise Manager or in the Collector and MOM in a cluster. A determining factor in your deployment decision can be the amount of heap memory that you can allocate to an Enterprise Manager. The available RAM and the JVM bit mode (32 or 64) that the underlying operating system supports determine the maximum amount of heap memory.
Deployment Guidelines for CA APM with CA CEM Monitoring
You can deploy CA CEM-only monitoring or CA APM for both agent and CA CEM monitoring. A CA CEM-only deployment has only TIM monitoring and no agent monitoring. CA CEM-only can be used for monitoring transactions in web applications that are not implemented using the Java and .NET technologies that agent monitoring support.
CA Technologies provides example deployment topologies along with associated provisioning recommendations to consider in your Introscope, CA APM, and CA CEM-only capacity planning. A table shows the components for each example deployment.
Number of TIMs
The number of TIMs you need depends on these factors:
  • Volume of traffic the TIM monitors
  • Number of different IP addresses the TIM monitors
  • Number of transactions being measured
  • Complexity or parsing the transactions
  • Quality of network packet delivery
  • Number of separate data centers
    Each data center with web traffic hosted on it needs at least one TIM.
Enterprise Manager Services Resource Usage
The Stats Aggregation service and the TIM Collection service can use significant resources. A well-designed CA APM deployment requires resources to support these two Enterprise Manager services. If the Collectors also collect agent metrics, the deployment requires additional resources.
Enterprise Manager Services and SmartStor
Enterprise Managers in a CA CEM-only deployment do not require a separate disk for SmartStor. Enterprise Managers running both Enterprise Manager services and an agent metric load must adhere to the SmartStor storage requirements.
TIM Collection Service
In a cluster one Collector hosts the TIM Collection service, which connects to all TIMs. The TIM Collection service cannot be distributed across multiple machines or multiple Collectors.
Stats Aggregation Service
A Collector hosting the Stats Aggregation service can appear to be under-utilized for large portions of any 24-hour period. The Stats Aggregation service consumes resources at the top of each hour for hourly stats aggregation. In addition, this service requires significant resources for a few hours beginning at 12:00 A.M. each night (assuming that the database contains at least 24 hours of statistics data).
JVM Sizes and Enterprise Manager Services
Each of the Enterprise Manager services has its own resource consumption characteristics.
For information about the heap requirements for Enterprise Manager services, see Example sizing for a clustered environment.
CA EEM
Using CA EEM is required for full functionality in CA APM deployments. Using CA EEM is optional when using Introscope for agent-only monitoring or CA CEM without access policies.
The resource usage for a CA EEM server is minimal when supporting a single cluster. If needed, you can colocate CA EEM on the same host as other components of a CA APM deployment, such as the computer hosting the MOM or APM database.
Follow these guidelines if your organization is using CA EEM to provide authentication services. Under the following conditions, host CA EEM on a separate computer:
  • For multiple CA Technologies products.
  • For a user population larger than the set of CA APM users.
If you follow the guidelines, CA EEM has no significant impact on Enterprise Manager capacity.
MOM
When running CA APM in a clustered environment, the MOM hosts the CEM console, as well as servicing all Workstation connections. For both CA CEM and Introscope monitoring, transactions are defined using the CEM console.
In planning any Introscope, CA APM, or CA CEM-only deployment, provide adequate resources for the MOM to allow capacity for additional dashboards and other client activity. Plan for growth as CA APM administrators and triagers discover new ways to use the product and expand enterprise monitoring.
Capacity Planning for Multiple Clusters
Introscope-only environments make relatively light use of the APM database server. In large installations multiple clusters can host their databases on a single PostgreSQL or Oracle database server. Be sure to configure separate databases on the server for each cluster and to provide adequate connections. Given sufficient computing resources, a single DBMS can support multiple CA APM clusters handling both Introscope and CA CEM data as well. However, CA CEM makes heavy use of the APM database. Monitor the database server to verify that it does not become overloaded.
The APM database and CA EEM are the only resources that can be shared across clusters. Other than these servers, the total sizing of your deployment is the sum of the individual cluster requirements.
The Cross-cluster Data Viewer (CDV) allows a single Workstation to view metric data from Collectors in up to 10 different clusters. To a Collector, a CDV connection is essentially another MOM connection. As such, it can significantly increase the query load on a Collector. If your Collectors handle multiple CDV connections provide additional CPU and memory resources, and be sure to follow the SmartStor optimization guidelines.
The CDV, APM database and CA EEM are the only resources that can be shared across clusters. Other than these servers, the total sizing of your deployment is the sum of the individual cluster requirements.
CA APM Sizing Tests
At a high level, your Enterprise Manager and cluster capacity are a function of these factors:
  • What you choose to monitor
  • Hardware resources that you provide
Since every application environment is different, general, one-size-fits-all recommendations are approximate and conservative. The best way to get an accurate picture of the resources you need for your CA APM environment is to execute load tests with the CA APM product in place. The test results can provide the values for you to input to the CA APM sizing tools.
Follow these guidelines for CA APM sizing tests:
  • Use external tools to measure CPU utilization, disk I/O, and network bandwidth utilization on the Enterprise Manager computer. For example, Windows Performance Monitor, vmstat, netstat, or esxtop.
  • Use the load test tools that you use for the general stability testing of your applications, and usage scenarios that are as typical as possible for your environment.
  • Run as many agents as you can with the resources available (up to the number expected in production).
  • Include all the nodes in your application topology.
  • Run the load for more than 72 hours, in order to see the full effect of SmartStor reperiodization and daily stats aggregation.
  • Include a client load. Use Workstations and WebView, as well as the CEM console. Look at dashboards and other data over various time periods.
    Note:
    You can capture and play back WebView traffic using most HTTP load generation tools.
    Note:
    Use the Identify Enterprise Manager Workload information to project resource requirements for higher loads.
  • Examine the Enterprise Manager logs for warnings and error messages. Comment text in the IntroscopeEnterpriseManager.properties file explains various logging options.
Enterprise Manager resource usage scales linearly. Divide resource measurements by the number of agents in your test run, and use these “per agent” values to project resource requirements for your production load.