Configure and Manage SmartStor Data

SmartStor stores metric and transaction data. The CA Introscope SmartStor database records all application performance data (CA Introscope metrics) always. This database information lets users analyze historical data, identify root causes of application downtime, or perform capacity analysis without the need for an external database.
apmdevops106
SmartStor stores metric and transaction data. The CA Introscope SmartStor database records all application performance data (CA Introscope metrics) always. This database information lets users analyze historical data, identify root causes of application downtime, or perform capacity analysis without the need for an external database.
2
2
SmartStor is enabled by default during CA Introscope installation. SmartStor data is set to age out over time, so the data store will not get excessively large. Multiple data files can grow in number as more data is generated.
Note:
For questions and answers about SmartStor, see TEC1651685.
SmartStor Data Storage
SmartStor records all application performance data (Introscope metrics) while it is in production, without the need for an external database. SmartStor lets you analyze historical data, identify the root causes of application downtime, or perform a capacity analysis. SmartStor is enabled with default settings when installing Introscope:
  • Once an hour, on the hour, SmartStor converts .spool files to .data files to optimize query processing.
  • Once a day, at midnight, SmartStor reformats historical metric data based on the Enterprise Manager tier settings and deletes aged out metrics and metadata.
SmartStor uses a storage hierarchy that assumes that recent data is accessed the most and has the highest resolution. The following data resolutions are based on the default property settings:
  • Store 1 week of data in 15-second chunks (uncompressed)
  • Store 30 days of data in minute (60 seconds) chunks (compressed)
  • Store 1 year of data in 15 minute (900 seconds) chunks (compressed)
SmartStor Data Files
The data files for SmartStor are:
  • The metadata directory-- Tracks information about the various metrics that the agents have reported. This directory tracks the names of the metrics.
  • Files with a .spool extension -- Holds the 15-second metric values sent to SmartStor since the last spool-to-query conversion on the Enterprise Manager. The .spool file never holds more than one hour worth of data. An example name is 1251226800000.spool.
  • Files with a .data extension -- Holds metric values in the read-only query optimized format in SmartStor. These files hold the data point values for 15 seconds, 1 minute, and 1 hour. An example name is 1251183600000.data.
By default the Enterprise Manager stores all SmartStor data in the
<EM_Home>
/data/metadata directory.
Warning!
Stop the Enterprise Manager before you back up or you work with the SmartStor data files. Copying or editing Enterprise Manager data files while the Enterprise Manager is running results in data loss.
When an Enterprise Manager is shut down, you can do all of the following tasks to back up and work with Enterprise Manager data:
  • Save data to backup media.
  • Edit SmartStor data using SmartStor tools.
  • Compress data.
  • Restore data.
  • Copy data for offline viewing by another Enterprise Manager.
Threshold for historical metrics is 30 mil.
You can change the threshold manually here:
\em\com.wily.introscope.em.feature\src\main\rootFiles\
config\apm-events-thresholds-config.xml
      <clamp id="introscope.enterprisemanager.metrics.historical.limit">
          <description>
              Per EM limit. Takes into account metrics with Smartstor data (i.e. live and historical metrics)
          </description>
          <threshold value="
30000000
"/>
      </clamp>
If you upgrade from APM 10.3 to APM 10.5 the threshold for historical metrics is not updated automatically.
Configure SmartStor Directory
Configure the directory that the Enterprise Manager uses for data storage.
Follow these steps:
  1. Go to the <
    EM_Home
    >/config directory.
  2. Open the IntroscopeEnterpriseManager.properties file in a text editor
  3. Enter values for the following properties:
introscope.enterprisemanager.smartstor.directory=data
introscope.enterprisemanager.smartstor.directory.metadata=data/metadata
introscope.enterprisemanager.smartstor.directory.archive=data/archive
introscope.enterprisemanager.smartstor.dedicatedcontroller=false
Determine SmartStor System Requirements
You use the APM Sizing Utility to determine the SmartStor system requirements. For more information, see APM Sizing Utility
Set Up a Dedicated Disk for SmartStor Data
Configure SmartStor to write to a dedicated disk. A dedicated disk avoids severely affecting Enterprise Manager performance and responsiveness. Disk contention with other Enterprise Manager activities or other processes running on the host can cause poor performance.
Follow these steps:
  1. Go to the <
    EM_Home
    >/config directory.
  2. Open the IntroscopeEnterpriseManager.properties file in a text editor.
  3. Set the following property to true:
    introscope.enterprisemanager.smartstor.dedicatedcontroller
  4. Save and close the file.
  5. Restart the Enterprise Manager.
Automatic Metadata Deletion
During the SmartStor reperiodization, Enterprise Managers automatically delete metadata (metric and agent descriptions) for which there are no associated metric data points. This process frees disk and memory space for reuse during SmartStor tiering.
Enterprise Managers do not delete historical agents that are manually mounted in a Workstation, including agents without metric data in SmartStor.
Configure SmartStor Tiers
The tiers in SmartStor define how long to store metric data. You can have a maximum of three tiers, each with the same or increasing frequency periods. The frequency represents the metric reporting granularity.
Each tier consists of a property pair:
  • Storage frequency (in seconds)
    • The frequency must be a multiple of 15 seconds.
    • The frequency must not be greater than 1800 seconds (30 minutes).
    • The frequency of each tier must be an even multiple of the previous tier frequency. This multiple can be one; each tier can have the same frequency.
  • Data aging period (in days). The age must not be zero.
Tier 1 is the minimum period. You configure Tier 2 based on the configuration of the frequency of Tier 1. For example, Tier 2 is as a summary of Tier 1, and Tier 3 is a summary of Tier 2. The following table shows how the tiers, frequency, and age relate:
Tier
Frequency in Seconds
Age in Days
1
15
7
2
15
14
3
30
10
  1. Nightly -- SmartStor collects metric data in an archive file, for example, a file named A1. A1 is stored in Tier 1.
  2. Day 8 -- A1 is promoted to Tier 2.
  3. Day 21 -- A1 is promoted to Tier 3. The frequency of data is converted to 30 seconds.
  4. Day 31 -- A-1 is removed from SmartStor.
Follow these steps:
  1. Go to the <
    EM_Home
    >/config directory.
  2. Open the IntroscopeEnterpriseManager.properties file in a text editor.
  3. Enter values for the following properties:
    introscope.enterprisemanager.smartstor.tier1.frequency
    introscope.enterprisemanager.smartstor.tier1.age
    introscope.enterprisemanager.smartstor.tier2.frequency
    introscope.enterprisemanager.smartstor.tier2.age
    introscope.enterprisemanager.smartstor.tier3.frequency
    introscope.enterprisemanager.smartstor.tier3.age
  4. Save and close the file.
  5. Restart the Enterprise Manager.
    If you later change your data tiers and restart the Enterprise Manager, SmartStor attempts to adjust data to your new configuration. SmartStor can lose up to one day of data.
Create a Query-only Duplicate Cluster
You can copy SmartStor data files to duplicate cluster machines to serve as a query-only source historical (non-live) data.
Note:
We recommend creating duplicate clusters only in production environments where the service demand considerably exceeds the capacity of the hardware on which the cluster is running.
Follow these steps:
  1. Create an exact duplicate of your production or test cluster. Use one computer for the MOM and one for each collector in your cluster.
  2. Shut down your production cluster.
  3. Copy the SmartStor, baselines, and Transaction Events data files from:
    • The production or test MOM to the duplicate MOM.
    • From each of the production or test collectors to the duplicate collectors.
  4. Start up both the production cluster and the query-only duplicate cluster.
  5. Run any desired queries on the query-only duplicate cluster.
  6. Repeat necessary steps once a day to keep current the data on the query-only duplicate cluster.
Use the SmartStor Command-Line Tools
The SmartStor tools let you perform different maintenance and administration tasks on the SmartStor database. These command-line tools that let you:
  • Clean up from metric leaks and explosions
    For information about metric leaks and explosions, see CA APM Sizing and Performance.
  • Split a single Enterprise Manager into multiple Enterprise Managers
  • Merge multiple Enterprise Managers into a single Enterprise Manager
Follow these steps:
  1. Shut down the source Enterprise Manager.
    CA Technologies recommends shutting down the source and destination Enterprise Managers when running SmartStor tools. Corruption can occur if the Enterprise Manager is writing data to SmartStor when SmartStor tools are also changing SmartStor data.
  2. Go to the
    <EM_Home>
    /tools/ directory.
  3. Open a command prompt and use one of the following start scripts:
    • Windows -- SmartStorTools.bat
    • UNIX -- SmartStorTools.sh
  4. Use the following syntax:
    <SmartStorTools start script> <command> <arguments>
    SmartStorTools uses the following arguments:
    • -agents -- A regular expression for agents
    • -dest -- Destination SmartStor directory
    • -help -- A help message
    • -metrics -- A regular expression for metrics
    • -silent -- Silent mode
    • -src -- Source SmartStor directory
    The special characters must be escaped using a backslash in the names of agents and metrics. The special characters are:
    ([{\^-$|]})?*+
    For example, specify the metric name SuperDomain|machineName|WebLogic|WebLogic as SuperDomain\|machineName\|WebLogic\|WebLogic.
    Warning!
    Use the remove and prune tools with caution because these tools delete data.
SmartStor Tool Commands
SmartStorTools cannot be running together with the Enterprise Manager process. Please check that Enterprise Managers is not running.
SmartStor provides the following tool commands:
  • merge
  • remove_metrics
  • remove_agents
  • keep_agents
  • prune
  • list-agents
  • test_regex
  • metadata_file
  • dump
Warning!
Use the prune tool with caution. Since this tool deletes data, take care that you do not accidentally delete important data.
merge
Merges a specified SmartStor source directory into a different destination directory. You can also use the merge command to migrate data to new APM versions, for example, from 10.5 to 10.7.
Arguments:
  • -dest - destination SmartStor directory
  • -help - prints a help message
  • -src - source SmartStor directory
Examples:
Windows:
SmartStorTools.bat merge <source directory> <destination directory>
For example:
SmartStorTools.bat merge <EM_Home>\data C:\data
When the command executes successfully, all the SmartStor data present in the source directory is merged with the data in the destination directory.
UNIX:
SmartStorTools.sh merge <source directory> <destination directory>
remove_metrics
Removes metrics matching the given regular expression. Use this tool to clean up metrics leaks and explosions.
Warning!
Use the remove_metrics tool with caution. Since this tool deletes data, take care that you do not accidentally delete important data.
Arguments:
  • -dest -- Destination SmartStor directory.
  • -help -- Prints a help message.
  • -metrics -- Regular expression for metrics to remove.
  • -src -- Source SmartStor directory.
Examples:
Windows
SmartStorTools.bat remove_metrics -dest <destination directory> -metrics <metrics to be removed> -src <source directory>
For example:
<EM_Home>\tools\SmartStorTools.bat remove_metrics -dest C:\SWDump\destination -metrics ".*Socket.*" -src <EM_Home>\data
When the command executes successfully, all the metrics except the socket metrics are present in the destination directory and the source directory has all the metrics intact.
UNIX
SmartStorTools.sh remove_metrics -dest < destination directory> -metrics <metrics to be removed> -src <source directory>
remove_agents
Removes agents matching the given regular expression.
Warning!
Use the remove_agent tool with caution. Since this tool deletes data, take care that you do not accidentally delete important data.
Arguments:
  • -agents -- Regular expression for agents to remove.
  • -dest -- Destination SmartStor directory.
  • -help -- Prints a help message.
  • -src -- Source SmartStor directory.
Examples:
Windows
SmartStorTools.bat remove_agents -agents <Agent name> -dest <destination directory> -src <source directory>
For example:
<EM_Home>\tools>SmartStorTools.bat remove_agents -agents ".*WebSphere.*" -dest C:\SWDump\destination -src <EM_Home>\data
When the command executes successfully, all the metrics related to WebSphere agents are removed from the destination directory and all the metrics in the source directory remain intact.
UNIX
SmartStorTools.sh remove_agents -agents ".*WebSphere.*" -dest <destination directory> -src <source directory>
keep_agents
Keeps agents matching the given regular expression
Arguments:
  • -agents -- Regular expression for agents to keep
  • -dest -- Destination SmartStor directory
  • -help -- Prints a help message
  • -src -- Source SmartStor directory
Examples:
Windows
SmartStorTools.bat keep_agents -agents <agent(s) name> -dest <destination directory> -src <source directory>
For example:
<EM_Home>\tools>SmartStorTools.bat keep_agents -agents ".*Websphere.*" -dest C:\SWDump\destination\keepagents -src data
When the command executes successfully, all the metrics related to Websphere agents are present in the destination directory and the contents of source directory are intact.
UNIX
SmartStorTools.sh keep_agents -agents <Agent(s) name> -dest <destination directory> -src <source directory>
prune
Finds agents that have no historical data in SmartStor, and removes all associated metric metadata. Use this tool to clean up metrics leaks and explosions.
Warning!
Use the prune tool with caution to avoid losing important data. The tool removes the metric definition and you lose access to historical data.
Arguments:
  • -dest -- directory containing pruned metadata file along with unchanged *.data files of src.
  • -help -- prints a help message
  • -silent -- silent mode
  • -src -- source SmartStor directory
Examples:
Windows
SmartStorTools.bat Prune -dest <destination directory> -src <source directory>
For example:
<EM_Home>\tools>SmartStorTools.bat Prune -dest C:\SWDump\destination\backup -src data
When the command executes successfully, metadata files are backed up to the
dest
directory with the name
metrics.metadata.
UNIX
SmartStorTools.sh prune -dest <destination directory> -src <source directory>
test_regex
Prints to the Windows or UNIX console the metrics or agents matching the given regular expression.
Arguments:
  • -help -- Prints a help message
  • -src -- Source SmartStor directory
  • -agents -- Regular expression for agents to match
  • -metrics -- Regular expression for metrics to match
Examples:
Windows
SmartStorTools.bat test_regex -agents <agent(s) name> -src <sourcedirectory>
For example:
<EM_Home>\tools>SmartStorTools.bat test_regex -agents ".*Websphere.*" -src <EM_Home>\data
When the command executes successfully, all the metrics for WebSphere agent are printed.
UNIX
SmartStorTools.sh test_regex -agents <agent(s) name> -src <source directory>
list_agents
Lists the agents matching a given regular expression.
The SmartStor tool list_agents tools list the agents matching the given regular expression.
Arguments:
  • -agents -- Regular expression for agents to match
  • -src -- Source SmartStor directory
  • -help -- Prints a help message
Examples:
Windows
SmartStorTools.bat list_agents -agents <agent(s) name> -src <source directory>
For example:
<EM_Home>\tools>SmartStorTools.bat list_agents -agents ".*Websphere.*" -src <EM_Home>\data
When the command executes successfully, all the agent names matching the regular expression (in this case ".*WebSphere.*") are listed with their last reported date.
UNIX
SmartStorTools.sh list_agents -agents <Agents name> -src <source directory>
metadata_file
Creates metadata file in format for pre 10.5 releases for a specified SmartStor directory (sourceDir) into file (destFile).
Arguments:
  • <source SmartStor directory>
  • <destination file>
Examples:
Windows:
SmartStorTools.bat metadata_file <source directory> <destination directory>
For example:
SmartStorTools.bat metadata_file <EM_Home>\data C:\data
UNIX:
SmartStorTools.sh metadata_file <source directory> <destination directory>
dump
Converts contents of metadata database to a set of CSV files.
Arguments:
  • <source SmartStor directory>
  • <destination directory for CSV files>
Examples:
Windows:
SmartStorTools.bat dump <source directory> <destination directory>
For example:
SmartStorTools.bat dump <EM_Home>\data C:\dump
When the command executes successfully, destination directory contains following files:
agents.csv, attributes.csv, folders.csv, metrics.csv, properties.csv
UNIX:
SmartStorTools.sh dump <source directory> <destination directory>
Redirect a Pruned Metric List to a Text File
You can redirect the Windows or UNIX pruned metrics list to a text file.
Follow these steps:
  1. Shut down the Enterprise Manager.
  2. From a command prompt, run the prune command and include the >> argument, for example:
    <
    EM_Home
    >\tools>SmartStorTools.bat prune -src <
    path to data directory
    > -dest D:\backup >>D:\console.txt
  3. Rename the metrics.metadata file in the <
    EM_Home
    >/data/metadata directory, for example: noprunemetrics.metadata.
  4. Run the test_regex command on the <
    EM_Home
    >/data/metadata directory.
    The Windows or UNIX console displays the command output showing all the metrics that are based on the regular expression you included in the command. Notice that the pruned metrics are no longer present.
    For example, if you were pruning many socket metrics, you run this test_regex command:
    On Windows:
    <
    EM_Home
    >\tools>SmartStorTools.bat test_regex -metrics ".*Socket.*" -src <
    path to data directory
    >
    On UNIX:
    <
    EM_Home
    >/tools>SmartStorTools.sh test_regex -metrics ".*Socket.*" -src <
    path to data directory
    >
  5. Restart the Enterprise Manager.
Trim SmartStor Data and Metadata
To trim excess SmartStor data and metadata, use the SmartStor tools.
Follow these steps:
  1. Shut down the Enterprise Manager.
  2. From a command prompt, run the test_regex command to print the current metric list.
    For example, if Introscope is installed in the <
    EM_Home
    > directory and SmartStor is configured to save data in the <
    EM_Home
    >\data directory, run this command:
    SmartStorTools.sh test_regex -metrics <
    agents regexp
    > -src ../data
    Example:
    SmartStorTools.sh test_regex -metrics ".*" -src ../data
    A list of fully qualified metric names appears. The total line count indicates the total number of metrics. However, the line count is not as accurate as the metric count you get from the supportability metrics in the Investigator.
  3. Compare line counts against specific regular expressions matching an agent or metric name. Determine what metrics occupy most of the space.
  4. From the list, pick metrics from specific agents to remove.
  5. Run the SmartStor tools remove_metrics command for the excess metrics.
    For example, if you have high SQL metric counts, use the following command:
    <
    EM_Home
    >\tools>SmartStorTools.bat remove_metrics -dest c:\sql_metrics_removed -metrics ".*SQL.*" -src <
    EM_Home
    >\data
    When the command executes successfully, all the metrics except the SQL metrics are present in the destination directory and the source directory has all the metrics intact.
    This command needs approximately as much target space to run as the space occupied by the original data (specified by the -src option). In the example, the new data without the SQL metrics (specified by the -dest option) is in the c:\sql_metrics_removed directory.
    This command is a long-running operation and can run for a few hours.
  6. Repeat the remove_metrics command on a different set of excess metrics.
    Sockets are another common source of leaked metrics. Use the following command to remove socket metrics:
    <
    EM_Home
    >\tools>SmartStorTools.bat remove_metrics -dest c:\sql_and_sockets_metrics_removed -metrics".*Sockets.*" -src c:\sql_metrics_removed
    This command uses the data that is derived from the first command that removed SQL metrics as the source data.. The data that has both SQL and socket metrics removed is in the -dest option location: c:\sql_and_sockets__metrics_removed. This command needs almost as much extra space to run as the original data.
  7. Move any backup metadata files from the <
    EM_Home
    >\data directory to a backup directory.
    The Enterprise Manager automatically uses the largest metadata file, which is the file with the largest number of defined metrics. Reducing metrics can result in backup metadata files being larger than the current data file.
Check the Metrics Data Value
Check the metrics data value for the cleaned metrics data and metadata after you trim excess SmartStor data and metadata.
Follow these steps:
  1. Verify that the Enterprise Manager is shut down.
  2. Open the apm-events-thresholds-config.xml file in the
    <EM_Home>
    \config directory. Note the current property setting, and then set the property:
    introscope.enterprisemanager.metrics.historical.limit=500000
  3. Save and close the file.
  4. Open the IntroscopeEnterpriseManager.properties file in the <
    EM_Home
    >\config directory. Note the current setting, and then set the property:
    introscope.enterprisemanager.smartstor.directory=<
    destination for the metrics removal
    >
    For example, the example destination is c:\sql_and_sockets_metrics_removed. This property setting is:
    introscope.enterprisemanager.smartstor.directory=c:\sql_and_sockets_metrics_removed
  5. Save and close the file.
  6. Restart the Enterprise Manager.
  7. Check the value of the following metric:
    Enterprise Manager | Connections | Number of Historical Metrics
    The value should be far below the original value. If the value is not lower, repeat the trimming steps. When the historical metric count is below 20 million, the Enterprise Manager runs more efficiently.
  8. (Optional) Reset the changed property values back to their original settings.
    Shut down the Enterprise Manager before resetting the values.
Example Use the test_regex Command to Find a SmartStor Data Problem
The text_regex command produces a readable version of metrics.metadata. In this example, you use the text_regex command to discover that JMX metrics data have exploded.
Follow these steps:
  1. Run the test_regex command, for example:
    <
    EM_Home
    >\tools>SmartStorTools.bat test_regex -metrics -src <EM_Home>\data
    The result is a list of the metadata metrics.
  2. Review the metadata metrics list and determine the probable source of the metric leak or explosion.
    For example, many metrics metadata entries can exist for JMX metrics.
  3. (Optional) Run a script to sort and group the metrics. For example, to verify that JMX metrics are the source of the metric explosion, run this script:
    grep '|'.* | awk -F'|' '{print $1}' | sort | uniq -c
    On UNIX, use the full test_regex command with a sort script:
    SmartStorTools.sh test_regex -metrics <
    agents regexp
    > -src ../data | grep '|'.* | awk-F'|' '{print $1}' | sort | uniq -c
    Example:
    SmartStorTools.sh test_regex -metrics ".*" -src ../data | grep '|'.* | awk -F'|' '{print $1}' | sort | uniq -c
  4. Review the sort results. For example, a list of metric counts and types appears:
    1054327 JMX
    1564 JSP
    1051 OlamWebApp
    889 EJB
    569 CEF
    496 Struts
    173 Edocs
    167 Servlets
    36 CPU
    29 Sockets
    In this case, SmartStor is storing over a million metrics and most are JMX metrics. This result indicates that a JMX metric explosion occurred. You can clean up the metrics using the SmartStor tools.
    For more information about metric explosions, see Java Agent or Microsoft .NET Agent.
Upgrade 8.x Data to 10.x
You can use a SmartStor Tools command to upgrade SmartStor data from 8.x agents to 10.x. The tool updates the 8.x frontend and backend metric IDs with 10.x frontend and backend metric IDs. Doing this upgrade prevents duplicate metrics from showing up in the Map display.
You use the upgrade command after upgrading the Enterprise Manager and the agent to 10.x.
Follow these steps:
  1. Shut down the Enterprise Manager.
  2. From a command prompt, browse to the
    <EM_Home>
    /tools directory.
  3. Back up the SmartStor data directory.
  4. Run the upgrade command. Use the following syntax and arguments:
    upgrade <sourceDir> <destDir>
    • sourceDir -- path to the existing directory containing SmartStor 8.x metrics
    • destDir -- path to a new directory
      The utility creates the directory. Do not create the directory in advance.
    Example:
    C:\Program Files\CA APM\Introscope<
    new_version(10.x)
    >\tools>SmartStorTools.bat upgrade C:\Program Files\CA APM\Introscope<
    new_version(10.x)
    >\data C:\Program Files\CA APM\Introscope<
    new_version(10.x)
    >\upgradeddata
  5. Open
    <EM_Home>
    /config/IntroscopeEnterpriseManager.properties in a text editor and locate the SmartStor Settings section.
  6. Set the value of introscope.enterprisemanager.smartstor.directory to the destination directory.
    The default value is data. This value refers to
    <EM_Home>
    /data/metadata.
  7. Restart the Enterprise Manager.