data_engine Release Notes

The data_engine probe manages and maintains data that is collected from monitoring probes that emit Quality of Service (QoS) data.
The data_engine probe manages and maintains data that is collected from monitoring probes that emit Quality of Service (QoS) data.
Support cases might not be viewable to all customers.
Revision History
(Included in UIM 20.3.3)
  • For Microsoft SQL Server 2016 (or higher), the existing data maintenance stored procedures have been enhanced to enable the truncation of partitions directly from the table. This enhancement eases the data maintenance process. For example, it eliminates the need of creating temporary tables, creating indexes, switching the partition data to the temporary tables, and then truncate the data. For Microsoft SQL Server 2012 or 2014, the existing stored procedures would continue to work as they were working earlier. That is, the new behavior is not applicable for these versions.
  • (9.20 HF5) Fixed an issue where data_engine was not processing all the qos_processor-enriched metrics after the restart.
  • (9.20 HF5) Fixed an issue where after updating the origin through qos_processor, data_engine was unable to store the related metrics.
  • Fixed an issue where users were unable to upgrade to 20.3.0. The upgrade was getting stuck at 40%. (Support Case: 32473364)
March 2021
(Included in UIM 20.3.0)
What's New:
  • (20.10 HF1) Fixed an issue which displays useful message in logs when the error code is thrown. (Support Cases: 20320491)
  • (20.10 HF1) Fixed an issue where after updating origin through qos_processor, data_engine cannot store their metrics. (Support Case: 31722751)
September 2020
(Included in UIM 20.1.0)
What's New:
  • (9.20 HF1) Fixed an issue in which after upgrading from 9.0.2 to 9.2, the data_engine probe 9.20 did not empty the hub queue due to initialization issues. (Support Cases: 20051358 and 20050695)
  • (9.20 HF2) Fixed an issue in which users were receiving the "You have exceeded your Qos License" error. (Support Cases: 20057740, 20051358, and 20050695)
  • (9.20 HF3) Fixed an issue in which data_engine was not adding the new metric definitions to the database. The issue was occurring because of the nan (not a number) value that was getting passed to the database. The database was then throwing exception and resetting the connection. This hotfix has resolved this issue by discarding nan before sending it to the database. (Support Cases: 20085489 and 20066349)
  • (9.20 HF4) Fixed an issue in which users were facing issue with the qos_processor queue when upgrading from 8.51 to 9.02. During an upgrade, data_engine runs a script that changes the checksum and uses SHA1 encoding to evaluate the checksum. Whereas,  qos_processor and data_engine are still evaluating the checksum based on SHA0, which causes the checksum mismatch and hence the issue. The hotfix resolves the issue by changing hashing in data_engine and qos_processor to use SHA1. (Support Cases: 20028603, 20089089, 20010894, and 20083614)
  • Fixed an issue related to data_engine probe which doesn’t start after upgraded to 9.20. (Support Case: 20051358)
  • Fixed an issues where data_engine is not adding new metric definitions to the database. (Support Case: 20085489)
  • Fixed an issue related to data_engine looping in initialization and the Hub queue is growing. (Support Case: 20050695)
  • Fixed an issue where GRE  QOS License exceeds license by 79897 objects. (Support Case: 20056526)
  • Fixed an issue where QOS is not showing for any device. (Support Case: 1369585)
  • Fixed an issue with the data_engine which is throwing multiple errors in the log and missing data in DB. (Support Case: 20066349)
March 2020
(Released with CA UIM 9 SP1)
What's New:
  • (9.02 HF1) Fixed an issue in which after you upgraded to CA UIM 9.0.2 in an environment where the back-end database server was Microsoft SQL Server 2008 R2, data_engine was experiencing failures even though the upgrade appeared successful. The following message was getting logged in the data_engine log file:
    (2) ExecuteNoRecords [Microsoft OLE DB Provider for SQL Server] 'CONCAT' is not a recognized built-in function name.
    You can also review the related KB Article.
  • (9.02 HF2) Fixed an issue in which when users were deleting a device through the USM and also selecting to delete the measurements, the QoS data was not getting deleted. To resolve the issue, this hotfix has reintroduced the probe utility (pu) command delete_qos in data_engine. This command was inadvertently removed from the product. (Support Case: 01222421)
  • (9.02 HF3) Resolved the following issues:
    • Fixed an issue in which data_engine was getting into a restart loop when a QoS was failing to insert. This was increasing the data_engine queue size. To resolve this issue, two parameters have been introduced: Continue_on_insert_fail and Ignore_errors_list. With these parameters, whenanyinsert fails, data_engine checks whether Continue_on_insert_fail is set to 1 and Ignore_errors_list includeslistof error messages that can be ignored. If these two conditions are met, data_engine continues with the insert. The default value of Continue_on_insert_fail is 1 and of Ignore_errors_list is “Invalid character value for cast specification | Unspecified error”. This hotfix is applicable when Microsoft SQL Server is used as the CA UIM database. (Support Cases: 01209184 and 01276828)
    • Fixed an issue in which data_engine was failing to insert QoS data into the database. This hotfix is more resilient to handle corrupt QoS messages. (Support Case: 01275045)
  • Fixed an issue where the following message was appearing in the Infrastructure Manager console. This message had no impact on the functionality and data_engine was working fine. (Support Cases: 01265179 and 01241208)
    NIS Data Engine: /***domain/***hub/***/data_engine - 11/19/18 19:55:51 - Information
    Database error: [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified - 11/19/18 19:55:52 - Error
    DB operation failed - 11/19/18 19:55:52 - Error
    Unable to connect to NIS Database - 11/19/18 19:55:52 - Warning
    No valid connection with NIS Database - 11/19/18 19:55:52 - Warning
  • Fixed an issue in which the snmpcollector probe was not storing data. This issue was resolved when the data_management_partition value was set to yes in the data_engine.cfg file. (Support Case: 01354166)
  • Resolved an issue in which the data_engine probe was not purging the old data because of the data_engine schedule setting was configured incorrectly. (Support Case: 01360960)
  • Fixed an issue in which data_engine alarms were failing to insert qos_data. The alarm (Insert bulk failed due to a schema change of the target table) was not honoring the alarm-severity configuration. This issue has been resolved by making updates so that the error condition is handled properly. (Support Case: 01355478)
  • Fixed an issue in which QoS was not showing up for any device. To resolve this issue, insert and data partitioning have been synchronized so that no two events happen simultaneously on the same table. (Support Case: 1369585)
  • Fixed an issue in which when users were trying to query the summarized QOS data in HN_ and DN_ tables, they were noticing that, for many CI's, the data was partial or was missing. (Support Case: 01254029)
  • Fixed an issue in which users were getting the following error (alarm) after every 10 minutes: (Support Case: 01293953)
    [Microsoft SQL Server Native Client 11.0] TCP Provider: An existing connection was forcibly closed by the remote host. Failed to insert QoS data into the database, check that the database is running.
  • CA UIM 9.2.0 has adopted OpenJDK 8u212 instead of Oracle JDK. Because of this change, CA UIM 9 SP1 (9.1.0) that was using Oracle JDK (JRE) 8u212 is no longer available and has been removed from the Support site. All the functionality that was included in 9.1.0 is now released as part of CA UIM 9.2.0. Consequently, all references to the 9.1.0 release and the probe version 9.10 (released with it) have also been removed from this probe documentation. We recommend that you move to this version 9.20 of this probe, as the previous version 9.10 is no longer available now. For more information about the OpenJDK usage in CA UIM, see  Adopting OpenJDK.
August 2019
Released with CA UIM 9.0.2
What's New:
  • Changed the default value of the following settings in the data_engine.cfg file:
    • data_management_timeout=65534
    • delete_raw_samples=35
    • delete_history_samples=180
    • thread_count_insert=2
  • Improved startup time and performance when using an Oracle database.
  • Added the update_metric_id=yes setting to data_engine.cfg. This setting automatically synchronizes the data between S_QOS_DATA and the CI_ tables. The update_metric_id setting instructs data_engine to update metric_id when it is changed. The S_QOS_DATA
    column can become outdated and can cause data to no longer be displayed in USM; this setting helps address this issue.
  • Added new configuration parameters for optimizing Oracle driver functionality:
    • Minimumnumber of sessions for the Oracle session pool.
    • Maximumnumber of sessions for the Oracle session pool.
    • Numberof prefetch rows for the Oracle driver.
    (Support Case: 00818890)
  • Added new configuration parameters for supporting Transport Layer Security (TLS) v1.2 for Oracle and Microsoft SQL Server databases.
  • Enhanced data_engine so that it also uses the
    value to uniquely decide whether the incoming QoS is a new one or an update of an existing QoS. When the first qos_message from a probe arrives, the data_engine probe adds QoS object data into the S_QOS_DATA table. The S_QOS_DATA table contains the object data for each unique combination of QoS, source, target, and origin attributes. Earlier, data_engine was using only QoS, source, and target attributes as a unique combination; it now uses the origin value also. This is helpful in case of multi-tenant scenarios.
    For example, before this enhancement, in a multi-tenant environment, consider a scenario where you have a primary hub and two separate hubs. The two separate hubs include robots with the same name and they are managing the same targets. When the primary hub (data_engine) receives QoS from these two separate hubs, the data_engine probe is not able to distinguish between the QoS coming from the first hub and the second hub. The combination is not unique in this case, as it includes the same QoS, source, and target. Now, with the implementation of this enhancement (inclusion of origin), data_engine correctly identifies the correct QoS coming from the specific hub, because it now uses the origin value, which forms a unique combination (QoS, source, target, and origin).
  • Updated this probe as part of removingdependencyon the end-of-life (EOL) Microsoft Visual C++ Redistributables in CA UIM 9.0.2. CA UIM 9.0.2 now uses Microsoft Visual C++ Redistributable for Visual Studio 2017.
  • Updated this probe as part of addressing CVE-2018-13820 and CVE-2018-13819 vulnerabilities in CA UIM 9.0.2.
Fixed Defects:
  • Fixed an issue where Microsoft SQL Server was consuming 100% CPU and memory. (Support Case: 953750)
  • Fixed an issue related to security vulnerabilities in  data_engine,distsrv, and ppm encryption. (Support Case: 1096152)
October 2018