Sizing Recommendations

This document provides information that is required to determine the sizing for an On-Prem APM deployment based on the current load. Follow the 3-step sizing process to determine your sizing requirement.
dxp10
Use the sizing recommendations to determine the sizing for an On-prem APM deployment. These recommendations are based on the current load of your environment. To determine the sizing requirements for your deployment, follow the sizing process. 
This section provides the following information:
 
2
 
Before you start with the sizing, we recommend reviewing the 
Deployment Architecture
 section.
Components Overview
The 
DX Platform
 runs on Kubernetes and consists of about 60 services. At a high level, the 
DX Platform
 deployment is divided into the following logical layers:
  • Application Service Layer
     
    Runs most of the DX Platform, DX APM,  DX OI services. This layer enables components to provide their functionality. Services are deployed to the available hardware using the Kubernetes orchestration layers based on available resources.
  • Elastic Service Layer
    Runs the Elasticsearch service and some services that are related to data storage and processing.
  • Kubernetes Master Layer
    Runs the Kubernetes master node. You are expected to provide a pre-configured Kubernetes environment.
The following diagram displays the
DX Platform
 components which are divided into logical groups: 
DXPlatform
Sizing the Environment
Compare the following assumptions to the specifications of your environment. If your environment is similar in size and load, then follow the sizing process that is described in this section. If your environment processes larger loads, or your environment has any other specific requirements, contact a 
Broadcom Customer Success Associate (CSA)
.
Assumptions
The following assumptions were made to calculate storage:
  • Every agent reports 2000 metrics.
  • Five agents are deployed per application.
  • Metrics are stored for:
    • 15 sec resolution - 7 days
    • 1 minute resolution - 23 days
    • 5 minutes resolution - 90 days
    • 15 minutes resolution - 365 days 
  • DX APM transaction traces are stored for seven days.
  • DX OI alerts are stored for seven days.
  • Each agent performs an average of four transaction traces per minute.
  • Average transaction trace size is 30 KB. 
Storage Descriptions
The DX Platform installation uses the following storage types:
Storage
Description
Elasticsearch Hot Storage
Reads and writes intensive storage that is used by Elasticsearch Hot nodes to store the latest data. This storage is locally attached to each Elasticsearch Hot node. Each Hot node has 2 TB of SSD storage locally attached.
Elasticsearch Warm Node
Reads extensive storage that is used by Elasticsearch Warm nodes to store older data. This storage is locally attached to each Elasticsearch Warm node. Each Hot node has 8 TB of SSD storage locally attached. 
NFS Storage
Stores the DX APM metrics and PostgreSQL data. This SSD NFS storage is attached during installation and uses a high-speed network.
Each storage has its own requirements that are listed in the 
Hardware and Software Requirements
 
section.
Mandatory Requirements
The following requirements are mandatory to implement these blueprints:
Requirement
Description
Elasticsearch Hot Node
Locally attached SSDs (NVMe preferred or high-end SATA SSD, IOPS - random 90K for read/write operations, throughput - sequential read 540 MB/s and write 520 MB/s, 4K block size, NIC 10 GB/s
Elasticsearch Warm Node
Locally attached or network storage with a minimum speed of 220 MB/s for read/write operations, 4K block size, NIC 10 GB/s
Kafka
Locally attached or network storage with a minimum speed of 220 MB/s for read/write operations, 4K block size, NIC 10 GB/s
Data Node
Locally attached or network storage with a minimum speed of 220 MB/s for read/write operations, 4K block size, NIC 10 GB/s
Kubernetes
Kubernetes 1.10.11 or 1.11.7, 1.12.5 or 1.13.2 installed on RedHat/CentOS 7.3, 7.5 or 7.6 systems, with Nginx Ingress Controller version 0.23.0 or above
Server
Each server has a 200-GB HDD to store temp files, docker:
  • /var/lib/docker - 150 GB
  •  /var/lib/kubelet - 45 GB
  • /var/log - 5 GB
File System
XFS, Support LVMs, No SWAP
Sizing Process
The sizing process helps you determine the 
storage, CPU, and memory
 requirements for the Application layer and the Elasticsearch layer. The memory requirements are determined based on the expected load of your environment.
The following section guides you through the sizing process for some common scenarios. If your requirement falls outside these scenarios, contact a 
Broadcom Customer Success Associate (CSA)
 for custom sizing.
Step 1: Determine Storage, CPU, and Memory Requirements
This section provides pre-calculated storage sizes for five tiers based on the number of agents for common use cases. 
DX APM Only
The following table displays the storage sizes for a DX APM-only installation:
Elasticsearch Layer
 
The following table displays the storage sizes for the Elasticsearch layer:
Number of Agents
< 1,500 (3 Mil Metrics)
< 4,500 (8 Mil Metrics)
< 8,000 (16 Mil Metrics)
< 16,000 (32 Mil Metrics)
< 50,000 (100 Mil Metrics)*
Elastic Hot Instances
3
3
5
9
14
Elastic Warm Instances
0
0
0
1
1
 
*
Average transaction trace for 50,000 agents has been lowered from 30 Kb to 15 Kb. 
 
Application Layer
 The following table displays the storage sizes for the Application layer: 
Number of Agents
< 1,500 (3 Mil Metrics)
< 4,500 (8 Mil Metrics)
< 8,000 (16 Mil Metrics)
< 16,000 (32 Mil Metrics)
< 50,000 (100 Mil Metrics)*
CPU (Cores)
68
109
160
292
830
Memory (GB)
159
237
356
654
1814
 
*
Average transaction trace for 50,000 agents has been lowered from 30 Kb to 15 Kb. 
 
Storage
The following table displays the storage sizes for NFS storage.
Number of Agents
< 1,500 (3 Mil Metrics)
< 4,500 (8 Mil Metrics)
< 8,000 (16 Mil Metrics)
< 16,000 (32 Mil Metrics)
< 50,000 (100 Mil Metrics)*
NFS Storage (GB)
600
1200
2000
3900
11800
Elasticsearch Hot (GB) per Instance
500
1500
1500
1700
1700
Elasticsearch Warm (GB) per Instance
-
-
-
400
2000
 
*
Average transaction trace for 50,000 agents has been lowered from 30 Kb to 15 Kb. 
 
DX APM and DX OI
The following table displays storage sizes for a DX APM installation with DX OI enabled:
Elasticsearch Layer
Number of Agents*
< 1,500 (3 Mil Metrics)
< 4,500 (8 Mil Metrics)
< 8,000 (16 Mil Metrics)
< 16,000 (32 Mil Metrics)
< 50,000 (100 Mil Metrics)*
Elasticsearch Hot Instances
 3
3
5
10
17
Elasticsearch Warm Instance
 0
0
0
0
0
The Elasticsearch nodes require locally attached storage.
 
*
Average transaction trace for 50,000 agents has been lowered from 30 Kb to 15 Kb. 
 
Application Layer
Number of Agents
< 1,500 (3 Mil Metrics)
< 4,500 (8 Mil Metrics)
< 8,000 (16 Mil Metrics)
< 16,000 (32 Mil Metrics)
< 50,000 (100 Mil Metrics)*
CPU (Cores)
140
184
229
348
903
Memory (GB)
307
393
492
742
1910
 
*
Average transaction trace for 50,000 agents has been lowered from 30 Kb to 15 Kb. 
 
Storage
Number of Agents
< 1,500 (3 Mil Metrics)
< 4,500 (8 Mil Metrics)
< 8,000 (16 Mil Metrics)
< 16,000 (32 Mil Metrics)
< 50,000 (100 Mil Metrics)*
NFS Storage (GB)
600
1200
1949
3820
11800
Elasticsearch Hot (GB) per Instance
500
1500
1600
1700
1600
Elasticsearch Warm (GB) per Instance
-   
-
-
-
-
*
Average transaction trace for 50,000 agents has been lowered from 30 Kb to 15 Kb. 
Step 2: Finalize and Determine the Deployment Specifications
Use the results from Step 1 to finalize the sizing blueprint. To determine the CPU and memory requirements for the Elasticsearch layer, use the number of instances from Step 1 and multiply this number by the fixed hardware requirements that are mentioned in the following table.
Server, Storage Type
The following table summarizes the sizing requirements:
Server Info
Cores
Memory
Storage Type
Kubernetes Layer
8-Core
16 GB
200-GB HDD non-OS locally attached
Application Layer
Total CPU from Step 1
Total Memory from Step 1
200-GB HDD non-OS local per server
Elasticsearch Layer (Per Instance)
  • Hot = (16-core) X Number of Hot instances from Step 1
  • Warm = (16-core) X Number of Warm instances from Step 1
  • Hot = (32 GB) X Number of Hot instances from Step 1
  • Warm = (32 GB) X Number of Warm instances from Step 1
  • Hot = Hot Storage per instance from Step 1
  • Warm = Warm Storage per instance from Step 1
Sizing Example
Use the following example to help you determine the sizing requirements for your DX APM deployment. Each step from the previous sizing process is highlighted in the example. In this example:
  • DX Application Performance Management and DX Operational Intelligence are enabled
  • 16,000 agents are considered
  • The default assumptions listed above are met
Step 1: Determine Storage, CPU, and Memory Requirements
The following table displays the storage details for 8000 agents:
APM Agents
16,000 Agents
Total Elastic Hot Instances
9 Instances
Elastic Hot Storage per Instance GB - SSD
1700 GB
Total Elastic Warm Instance
1 Instance
Elastic Warm Storage per Instance GB - SSD
400 GB
NFS Storage (GB) - SSD
3900 GB
Application Layer - CPU (Cores)
292 Cores
Application Layer - Memory (GB)
654 GB
Step 2: Finalize and Determine the Deployment Specifications
The systems that are available for the Application layer are of 24 cores and 64 GB of memory. We do not recommend using systems with less than 16 cores and 32 GB RAM. Continuing with the example for 16,000 Agents, you must provision the following specifications: 
CPU and Memory Specifications by Server
Server Info
Total Cores
Total Memory (GB)
Number of Instances Based on Total CPU and Memory
Kubernetes Layer 
8
16
1 Instance
Application Layer 
292
654
13 instances 
of 24-core, 64 GB per server
Elasticsearch Hot
144
288
9 Instances 
of 16-core, 32 GB per server
Elasticsearch Warm
16
32
1 Instance 
of 16-core, 32 GB per server
CPU and Memory Specifications by Configuration
Configuration
Instances
Purpose
24-core, 64 GB
13
Application Layer
8-core, 16 GB
1
Kubernetes Master Layer
16-core, 32 GB
9
Elasticsearch Hot
16-core, 32 GB
1
Elasticsearch Warm
Storage Specifications by Server
Server Info
Size (GB)
Partition Label
Type
Number of Instances
Comments
Kubernetes Layer
200
APP_LOCAL
HDD
1
Kubernetes orchestration
Application Layer
200
APP_LOCAL
HDD
13
System and temporary files
 
3900
NFS_SHARE
SSD
1
NFS share storage
Elasticsearch Hot
2048
ES_HOT
SSD
9
Elasticsearch Hot storage
 
200
APP_LOCAL
HDD
9
System and temporary files
Elasticsearch Warm
512
ES_WARM
SSD
1
Elasticsearch Warm storage
 
200
APP_LOCAL
HDD
1
System and temporary files
Storage Specifications by Type
Type
Partition Label
Instances
Size
Purpose
HDD
APP_LOCAL
24
200 GB
System and temporary files
SSD
ES_HOT
9
2 TB
Elasticsearch Hot Storage
SSD
ES_WARM
1
512 GB
Elasticsearch Warm Storage
SSD
NFS_SHARE
1
3900 GB
Metric data and config
Cloud Proxy Sizing
Cloud Proxy forwards APM Isengard communication to DX APM cluster using the WebSocket protocol. It also acts as a point of presence for complex environments. Cloud Proxy is an optional component and is deployed separately. For more details, see the 
Cloud Proxy 
documentation.
If you are using the Cloud Proxy, consider the following sizing requirements:
  • The Cloud Proxy requires 3 CPUs and 6-GB of memory. The Cloud Proxy can serve 5000 agents with a normal payload size. You can horizontally scale the Cloud Proxy by deploying multiple proxies to the same instance.
  • The host operating system must have the maximum number of open file descriptors per-process with the system resources limit set above the minimum 16384.
    • Two sockets are required for each Isengard agent connection. Three sockets are required per each HTTP/WS agent connection due to the loop back to the Isengard server.
Sizing Limitations
The maximum number of agents is 5000 or 10 million metrics per tenant.