data_engine

Data that is produced by monitoring probes is stored in the UIM database, in tables named raw data tables (table prefix is RN_). Raw data is kept for a user-defined period, then summarized and aggregated into Hourly data (HN_tables). Hourly data is then summarized and aggregated into Daily data (DN_ tables).
uimpga-ga
HID_data_engine
The data_engine probe manages and maintains data that is collected from monitoring probes that emit Quality of Service (QoS) data. The data_engine creates all tables and stored procedures necessary to manage the collected data.
Data that is produced by monitoring probes is stored in the UIM database, in tables named raw data tables (table prefix is RN_). Raw data is kept for a user-defined period, then summarized and aggregated into Hourly data (HN_tables). Hourly data is then summarized and aggregated into Daily data (DN_ tables).
The data_engine probe does not generate alarms and has no metrics.
This article provides information for the following topics :
Tables
The tables created in the UIM database have prefixes indicating the type of data they contain.
The naming convention for the tables is as follows:
  • S_ for tables that are used to store system data
  • D_ for data tables
  • H_ for tables containing historic data
  • BN_ for data tables containing baseline data
  • HN_ for data tables containing hourly/compressed data
  • DN_ for data tables containing daily/compressed data
  • RN_ for data tables containing unprocessed (raw) data directly from the probes
The QoS data structure is dynamically created by the data_engine on the first startup, and when the first unique QOS_DEFINITION or QOS_MESSAGE message is received from a probe. The S_QOS_DEFINITION table contains the definitions of known QoS types (for example, QOS_CPU_USAGE), and is updated when a probe sends a QOS_DEFINITION describing a new QoS type.
The S_QOS_DATA table contains an index of all data tables for the QoS objects. When a probe sends a QOS_MESSAGE containing a QoS object that is not defined in the S_QOS_TABLE, the following actions occur:
  • A new entry is added to the table.
  • Data is inserted into the table referenced in column r_table (typically RN_QOS_DATA_nnnn) with the table_id that the new row is given when inserted into the S_QOS_DATA table.
Do not drop the data tables manually. Instead delete the entry from the S_QOS_DATA table, and the tables are dropped by a trigger. Restart the data_engine afterwards.
How the data_engine Probe Collects and Maintains QoS Data
During data collection, data_engine performs the following actions:
  1. The data_engine probe receives a QoS definition from another probe.
  2. The data_engine probe queries the S_QOS_DEFINITION table to determine if the QoS type in the message already exists in the table (for example, QOS_CPU_USAGE).
  3. If the QoS type does not exist, a new entry is added to the S_QOS_DEFINITION table. New RN, HN, BN and DN tables are created in the UIM database to store monitoring data from the probe.
  4. When the first qos_message from the probe arrives, the data_engine probe adds QoS object data into the S_QOS_DATA table. The S_QOS_DATA table contains the object data for each unique combination of origin, QoS, source, and target attributes.
    Note:
    For data_engine released with versions prior to CA UIM 9.0.2, the S_QOS_DATA table contains the object data for each unique combination of QoS, source, and target attributes. It does not include the origin value in identifying the unique combination. However, the data_engine probe in CA UIM 9.0.2 has been enhanced to use the unique combination of QoS, source, target, and origin attributes. Therefore, when you upgrade from a previous version to CA UIM 9.0.2, the complete upgrade process might take some time because data_engine uses the origin value to recalculate the unique combination. This recalculation results in updating the existing values in the checksum column of the S_QOS_DATA table. The data_engine probe runs database scripts during the upgrade process to update the existing records. For example, for Microsoft SQL Server and MySQL, the respective scripts take approximately 20-30 seconds to update 700,000 records in the S_QOS_DATA table. In this case, the computer has 16 GB of memory and 8 cores processor.  
  5. The data_engine probe inserts the raw data from the probes into the appropriate RN tables.
  6. During the scheduled maintenance runs, the data_engine probe summarizes and aggregates raw data from RN tables into hourly data that is stored in HN tables.
  7. Older RN data is purged based on a user-defined period.
  8. The data_engine probe then summarizes and aggregates hourly data from HN tables into daily data that is stored in DN tables.
  9. Older HN and DN data is purged based on a user-defined period.
  10. Last sample value coming from the probes for each QoSobject is populated in the S_QOS_SNAPSHOT table. This data is used to provide fast QoS data access for CA Unified Management Portal (UMP) portlets.
RN_QOS_DATA Table Columns
RN_QoS_Data_tables hold raw QoS data. QoS data is written once and never updated.
Column Name
Description
TableID
unique identifier; key for looking up timeseries data
Sampletime
time the sample was taken
Samplevalue
QoS value
Samplestdev
standard deviation of the sample
Samplerate
Rate of sampling
Tz_offset
time zone offset
RN_table Indexes
The default indexes in RN_tables are optimized for writing data:
Index
Description
Idx0
sampletime, table_id
Idx1
table_id, sampletime (with samplevalue, samplerate, tz_offset included in the index)
The RN_QoS_DATA_tables do not have primary keys, as both tableID and sampletime can be duplicated.
Data_engine Start Up
By default, the data_engine commits data to a database in parallel mode using ten parallel threads. In parallel mode, the data_engine uses multiple threads to do work in parallel and makes the data_engine less vulnerable to performance issues when commits to a particular table are slow. For more details, see Parallel ModeSerial Mode, and Thread Count.
When the data_engine probe starts, it loads both S_QOS_DATA and S_QOS_DEFINITION into memory and establishes a bulk connection to the database.
Parallel Mode
The data_engine probe can run in either parallel or serial mode. The mode is determined by the thread count, which is the number of preallocated threads used to commit data to a database. Earlier versions of data engine ran in serial mode by default, but the current default is parallel mode. When the thread_count_insert parameter in Raw Configure is set to a value of greater than one, the data_engine commits data to the database in parallel mode; when set to a value less than one serial mode is used.
The default setting for the thread_count_insert parameter is 10.
In 
parallel
 mode, the data_engine:
  1. Continuously reads data from the hub.
  2. Stores the data in a shared list and begins reading more messages.
  3. Iterates over all lists using another thread to see if any of the lists should be flushed to the database.
    The actual writing of data does not happen now.
  4. All objects are marked as ‘ready to commit’ and a reference is placed onto another list.
  5. Concurrently, a thread continuously runs to validate, sort, and place messages into a list that is written to the database.
  6. Writes any messages that have been marked as 'ready to commit data' to the database using a thread pool of worker threads.
Serial Mode
Serial mode is the original mode for the data_engine probe. If the thread_count_insert parameter is set to zero in the Raw Configure menu, the data_engine probe defaults to serial mode.
In this mode, the data_engine:
  1. Reads messages from the hub for a given time period (default around 1 second).
    • From 1 through 20 messages are read at a time, depending on how many are in the queue or if the hub queue size has changed (default 20).
    • The read messages are then validated and sorted into lists that can be quickly inserted into the database.
  2. Stops reading new messages and iterates over all the lists, checking to see if any are full. By default the list is flushed if it contains more than 5000 messages or if it has not been flushed in the last 5 seconds.
  3. Goes back to reading messages from hub. If one bulk object takes too long to insert, then the writing of all data to the database is delayed.