Data Science Platform Pod

 uses a proprietary data science engine that applies statistical process controls to identify significant deviations from typical system behavior. CA Digital Operational Intelligence builds an Analytics Chart to depict the most typical values of a metric as a center line. This line is bounded by colored bands above and below, that track the characteristic volatility of the metric over time. Alert points are identified where the analysis engine detects an unusual event relative to the historical data. Alerts are grouped by their maximum deviations from typical behavior.
doi13
Digital Operational Intelligence
 uses a proprietary data science engine that applies statistical process controls to identify significant deviations from typical system behavior. CA Digital Operational Intelligence builds an Analytics Chart to depict the most typical values of a metric as a center line. This line is bounded by colored bands above and below, that track the characteristic volatility of the metric over time. Alert points are identified where the analysis engine detects an unusual event relative to the historical data. Alerts are grouped by their maximum deviations from typical behavior.
Data Science Platform Container Environment Variables
The following table describes the following environment variables of the Data Science Platform Container:
Environment Variable
Description
DB_SERVER
Specifies the hostname for the Postgres database, which is installed with CA Digital Operational Intelligence.
DB_PORT
5444
DB_USER
Specifies the name of the database user for CA Digital Operational Intelligence. 
Default Value:
 dspuser
DSP_CRON
When this variable is set to ‘True’, Data Science Platform schedules cron jobs to run ‘nightly model build’, ‘convert raw data into condensed binary data’ and ‘prune old data’ scripts. On a multi-container environment, only one container should have this field set to ‘True’. Otherwise, competing jobs can corrupt the data.
Default Value:
True
TZ
Specifies the time zone
Default Value:
UTC
DB_CLEAR_PASS
An alternative to the encrypted password (which is the default in containers). In this field, you can pass the clear password for the database and Data Science Platform auto encrypts and saves in the environment file. You need not use this field.
DB_NAME
Specify the name of the database where the CA Digital Operational Intelligence Platform is installed.
DSP_DUMP_GH
When this variable is set as True, the Data Science Platform dumps out the Green Highway values every night and it writes the output to a mounted file system.
KDE_THREAD_COUNT
Specifies the number of threads
DSP_TIME_FORMAT
Indicates the time format
DSP_SERVER_PORT
Specifies the Data Science Platform port number
DB_RECONNECT_TRIES
Indicates the number of attempts to be made to reconnect to the Postgres database
Default Value:
30
DB_RECONNECT_DELAY
Indicates the interval in seconds between attempts to reconnect to Postgres
Default Value:
100
Data Science Platform Integrator Environment Variables (DSPIntegrator Container) 
The following table describes the following environment variables of the DSPIntegrator Container:
Environment Variable
Description
DSP_SERVER_HOSTNAME
Specifies the hostname of DSPCASA container
Default Value:
localhost
DSP_SERVER_PORT
Specifies the Data Science Platform Server port number
Default Value:
58080
DSP_BULK_SIZE
Number of messages that are sent to the Data Science Platform per request
Default Value: 2500
DSP_LOG_LEVEL
The log level for Data Science Platform Integrator 
Default Value: info
PROCESSING_THREAD_POOL_SIZE
Indicates Number of threads utilized for processing
Default Value
: 64
BLOCKING_Q_CAPACITY
Indicates the Maximum number of records stored in Processing Queue
Default Value:
250000
CONSUMER_THREADS
Displays the number of threads reading data from Kafka
Default Value:
2
KAFKA_GROUP_ID
Indicates Kafka Group ID for Consumer 
Default Value:
dsp_integrator
KAFKA_ENABLE_AUTO_COMMIT
Kafka consumer property to enable auto commit
Default Value:
true
KAFKA_AUTO_OFFSET_RESET
Kafka consumer property to set offset reset 
Default Value:
latest
KAFKA_AUTO_COMMIT_INTERVAL_MS
Kafka consumer property to set commit interval in milliseconds
Default Value:
10000
KAFKA_SESSION_TIMEOUT_MS
Kafka consumer property for session time out in milliseconds
Default Value:
30000
KAFKA_METADATA_MAX_AGE_MS
Kafka consumer property for metadata data max age in milliseconds.
Default Value:
10000
KAFKA_MAX_PARTITION_FETCH_BYTES
Kafka consumer property for max fetch size for records in a partition.
Default Value:
31457280
JARVIS_TOPIC_PREFIX
Topic prefix values for topic name in Jarvis.
Default Value:
CAA_DTR__SV_BL__,CAA_DTR__SV__
PROCESSING_WAIT_TIME
Time interval between consecutive reads from Kafka (Approx processing time)
Default Value:
5000
KAFKA_MAX_POLL_RECORDS
Max number of records that can be polled in one read
Default Value:
10000
KAFKA_MAX_REQ_SIZE
Kafka consumer property for maximum bytes in a request.
Default Value:
5000000
ES_SOCKET_TIMEOUT
Elastic search connection socket timeout in seconds
Default Value:
30000
ES_CONNECT_TIMEOUT
Elastic search connection timeout in seconds
Default Value:
5000
ES_MAX_RETRY_TIMEOUT
Elastic search connection max retries timeout in seconds.
Default Value:
60000
ES_PROTOCOL
Elastic search protocol for communication
Default Value:
http
JARVIS_METADATA_TOPIC
Indicates the name of Jarvis meta data topic
Default Value:
jarvis_metadata
METRIC_INDEX_ES
Specifies the Elasticsearch indexes to monitor (comma separated index names)
Default Value:
ao_itoa_metrics_uim_1*,ao_itoa_metrics_ada_1*,ao_itoa_metrics_nfa_1*,ao_itoa_metrics_service_sa_1*,ao_itoa_metrics_custom_1*,ao_itoa_metrics_apm_1*
METRIC_INDEX_ES_CAPM
Specifies the CA Performance Management metric index names to monitor (comma separated index names)
Default Value:
ao_itoa_metrics_capm_availability_1*,ao_itoa_metrics_capm_reachability_1*
CI_NAME_FIELD
Field name from Each Product Metrics are stored as CI Field in Anomaly Alarms , first field with values gets picked for each Product.
Default Value:
ci_name, configuration_item,agent
CI_TYPE_FIELD
Field name from Each Product Metrics are stored as CI Type Field in Anomaly Alarms , first field with values gets picked for each Product.
Default Value:
ci_type, configuration_item_type
CI_ID_FIELD
Field name from each Product Metrics are stored as CI ID Field in Anomaly Alarms , first field with values gets picked for each Product.
Default Value:
ci_id, ci_unique_id, itemName
ALARM_SEVERITY
Specifies the severity of an alarm. The severity can be critical,major,minor
METRICS_CAPM_<
metric name
>
Specifies the metric field names to monitor for each index name that is specified in METRIC_INDEX_ES_CAPM.
Each field corresponds to the metric field in the index separated by a comma.
At present,
this variable does not support metric fields that have a null value.
Default Value:
METRICS_CAPM_AVAILABILITY=availability
METRICS_CAPM_REACHABILITY=reachability
UIM_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for CA Unified Infrastructure Management
Default Value:
5
CAPM_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for CA Performance Management
Default Value:
5
ADA_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for CA Application Delivery Analysis
Default Value:
5
NFA_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for CA Network Flow Analysis
Default Value:
5
SA_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for Service Alarms
Default Value:
5
APM_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for
CA Application Performance Management
Default Value: 1
CUSTOM_POLL_INTERVAL
The poll interval (in minutes) used to send data to the Data Science Platform for a custom product
Default Value:
5
UIM_EXCLUDE_METRIC
Specifies the metric field name to be excluded for CA Unified Infrastructure Management
CAPM_EXCLUDE_METRIC
Specifies the metric field name to be excluded for CA Performance Management
ADA_EXCLUDE_METRIC
Specifies the metric field name to be excluded for CA Application Delivery Analysis
NFA_EXCLUDE_METRIC
Specifies the metric field name to be excluded for CA Network Flow Analysis
SA_EXCLUDE_METRIC
Specifies the metric field name to be excluded for Service Alarms
APM_EXCLUDE_METRIC
Specifies the metric field name to be excluded for
CA Application Performance Management
CUSTOM_EXCLUDE_METRIC
Specifies the metric field name to be excluded for a custom product
UIM_DOCTYPE_VER
Specifies the Doc Type Version
Default Value:
1
CAPM_DOCTYPE_VER
Specifies the Doc Type Version
Default Value:
1
NFA_DOCTYPE_VER
Specifies the Doc Type Version
Default Value:
1
ADA_DOCTYPE_VER
Specifies the Doc Type Version
Default Value:
1
APM_DOCTYPE_VER
Specifies the Doc Type Version
Default Value: 1
ALARM_DOCTYPE_VER
Specifies the Doc Type Version
Default Value:
1
CRITICAL_ALARM_MSG
Specifies the alarm message for Anomaly Alarms
Default Value:
"Rare Anomaly (outside 99.7th percentile probability range) detected"
MAJOR_ALARM_MSG
Specifies the alarm message for Anomaly Alarms
Default Value:
"Probable Anomaly (outside 95th percentile probability range) detected"
MINOR_ALARM_MSG
Specifies the alarm message for Anomaly Alarms
Default Value:
"Common Anomaly (outside 68th percentile probability range) detected"
DISABLE_EWMA_ALARMS
Indicates if the anomaly alarm raised by the Data Science Platform is enabled or disabled for the EWMA model type
Default Value:
true
KAFKA_BROKERS
Kafka servers & port comma separated
Default Value:
kafka1:9092
INGEST_TOPIC
Indicates the topic name that is ingested in Jarvis.
Default Value:
CAA_unverified_p1
KAFKA_BATCH_SIZE
Kafka producer property , sets max records to be posted in a batch
Default Value:
1000
KAFKA_LINGER_MS
Kafka producer property , time interval before posting data in Kafka
Default Value:
100
KAFKA_ACKS_CONFIG
The number of acknowledgments the producer requires the leader to have
received before considering a request complete.
Default Value:
1
KAFKA_PRODUCER_TIMEOUT_MS
The configuration controls the maximum amount of time the client will wait
for the response of a request
Default Value:
30000
MIN_MEMORY_LIMIT
Indicates the minimum memory limit of DSP integrator Java heap memory.
Default Value:
2g
MAX_MEMORY_LIMIT
Indicates the maximum memory limit of DSP integrator Java heap memory.
Default Value:
6g
NFA_METRIC_NAMES
Defines the complete set of CA Network Flow Analysis metric names
Default Value:
totalbytes,totaltransactiontimemax,icmp,serverresponsetimemin,waasoutputbytes,totaltransactiontimemin,packetsretransm,outpkts,serverresponsetimemax,totaltransactiontimesum,
servernetworktimemin,concurrentsessions,inbytes,servernetworktimesum,clientnetworktimesum,inpkts,lateresponses,outbytes,serverresponsetimesum,tcpflags,waasinputbytes,
clientnetworktimemax,artnewconnections,clientnetworktimemin,servernetworktimemax,tos,serverpacketsretransm,dsttos,newconnections,waassegment,waaspassthroughreason,clientpackets,
transactioncount
ALARM_THRESHOLD
Indicates the threshold at which various alarm types are generated
Reference values for the Alarm Threshold variable are:
  • 0
    : No anomaly alerts are generated
  • 1
    : Only critical anomalies are generated when the upper threshold for the Rare band is crossed
  • -1
    : Only critical anomalies are generated when the upper and lower thresholds for the Rare band is crossed
  • 2
    : Critical and major anomalies are generated when the upper threshold value for Rare and Probable bands are crossed
  • -2
    : Critical and major anomalies are generated when the upper and lower threshold values for Rare and Probable bands are crossed
  • 3
    : Critical, major, and minor anomalies are generated when the upper threshold value for Rare, Probable and Common bands are crossed
  • -3
    : Critical, major, and minor anomalies are generated when the upper and lower threshold values for Rare, Probable and Common bands are crossed
Default Value:
3
ENABLE_CAPM_METRICS
If this environment variable is set to true, probability bands appear for CA Performance Management metrics.
Default Value:
false
ADA_NETWORK_METRIC_NAMES
Defines the complete set of
CA Application Delivery Analysis
Network metric names
Default Value:
averageNRTT,averageRetransDelay,averageNCT,averageEffectiveRTT 
ADA_SERVER_METRIC_NAMES
Defines the complete set of
CA Application Delivery Analysis
Server metric names
Default Value:
averageDTT,averageTTT,averageSCT,averageSRT,refusedRatio,unresponsiveRatio
APM_METRIC_NAMES
Defines the complete set of
CA Application Performance Management
metric names
Default Value:
active_connection_object_count,active_threads,active_user_sessions,amount_of_space_used_bytes,availability,average_browser_render_time_ms,
average_connection_establishment_time_ms,average_dns_lookup_time_ms,average_dom_construction_time_ms,average_dom_processing_time_ms,
average_domain_lookup_time_ms,average_method_invocation_time_ms,average_page_load_complete_time_ms,average_page_load_time_ms,
average_page_render_time_ms,average_page_stall_time_ms,average_previous_page_unload_time_ms,average_resource_load_time_ms,
average_response_download_time_ms,average_response_time_ms,average_result_processing_time_ms,average_round_trip_time_ms,
average_time_to_first_byte_ms,average_time_to_last_byte_ms,buffer_busy_waits,bytes_in_use,bytes_total,callback_execution_time_ms,
concurrent_invocations,current_capacity_bytes,database_buffers,database_cpu_time_ratio,database_wait_time_ratio,defects_per_interval,
errors_per_interval,fixed_size,free_buffer_inspected,gc_invocation_total_count,gc_invocations_per_interval_count,gc_time_per_interval_ms,
get_count,growth_rate_bytes_second_over_past_1_minute,invocation_count_per_interval,maximum_capacity_bytes,memory_sorts_ratio,
no_of_logons,page_hits_per_interval,percentage_of_java_heap_used,percentage_of_maximum_capacity_currently_used,
percentage_of_time_spent_in_gc_during_last_15_minutes,physical_read_total_bytes_per_sec,physical_reads,
physical_write_total_bytes_per_sec,physical_writes,redo_allocation_retries,redo_buffers,redo_entries,response_download_time_ms,
response_time_95th_percentile_ms,response_time_per_txn,responses_per_interval,soap_faults_per_interval,soft_parse_ratio,
sort_memory,stall_count,time_to_first_byte_ms,total_availability_defective_transactions_per_interval,total_bt_group_instances_per_interval,
total_defects_per_interval,total_gc_time_ms,total_memory,total_performance_defective_transactions_per_interval,total_resource_load_time_ms,
total_rt_95th_percentile_transactions_per_interval,total_session_count,total_sessions_usage,total_sga,total_transactions_per_interval,
utilization_aggregate,utilization_process,variable_size,wait_count,average_block_time_ms,average_bytes_allocated,
average_response_time_ms_deviation,average_response_time_ms_prediction,average_response_time_ms_variance_intensity,
average_system_cpu_time_ms,average_user_cpu_time_ms,average_wait_time_ms,connection_count,cpu_rounded,defect,
estimated_message_processing_time_ms,memory_rounded,messages_received_per_interval,packets_received,packets_sent,
responses_per_interval_for_vivo,responses_per_interval_for_wifi,total_defect_ratio,inoctets,outoctets,totaluserpercent,
totalsystempercent,totalutilpercent,meminusepercent,used_blocks,used_space_in_database_blocks,maximum_size_in_database_blocks,used_space,
changes_sent_per_interval,totaidlepercent,totalstealpercent,memoryinusebytes,memoryinusepercent,swapinusebytes,swapinusepercent,
dropbytesrate,errorbytesrate,sendreceivebytesrate,readbytesrate,writebytesrate,diskinusebytes,diskinusepercent,loadaverage15min,loadaverage1min,loadaverage5min
Alarm Clustering Environment Variables (AnalyticsJob Container)
Environment Variable
CLUSTER_NAME
ElaticsearchCluster Name
Default Value:
digital_oi
ES_SERVER
Hostname of the Elastic Search server      
Default Value:
elasticsearch
ES_PORT
Port number of Elasticsearch on which the service is running
Default Value:
9200
TENANT_ID
Specifies the tenant name.
ES_CLUSTER
The hostname of Elastic Search Cluster     
Default Value:
elasticsearch         
ES_CLUSTER_HTTP_PORT
Elastic search Cluster Http Port value
Default Value:
9200
SERVICE_ALARM_GROUPING_TIME
Alarms generated for a service within an interval of time that belongs to the same service alarm
Exmaple:
300
JARVIS_INGESTION_HOSTNAME
Jarvis hostname by which ingestion is to be done 
Default Value:
apis
JARVIS_INGESTION_PORT
Jarvis port number on which the service is running
Default Value:
8080
KAFKA_HOSTNAME
Specifies the Kafka server hostname or internal container name
Default Value:
localhost
KAFKA_CLUSTER
Kafka Cluster hostname and port
Default Value:
kafka:9092
MIN_MEMORY_LIMIT
Minimum memory allocated to container
Default Value:
1g
MAX_MEMORY_LIMIT
Maximum memory allocated to container
Default Value:
2g
KAFKA_PORT
Specifies the Kafka port number
Default Value:
9092
READSERVER_HOST
Indicates the
Digital Operational Intelligence
readserver host name
Default Value:
readserver:8080
ALERT_TOPIC_PRIORITY
Priority supported for the alarm topics
Example:
P1
ALERT_TOPIC_NAMES
Indicates the kafka alert topic priority
Default Value:
ao_itoa_alarms_all_1_
PREDICTION_INDEX_ES
Specifies the Prediction indexes to monitor (comma separated index names)
Default Value:
ao_itoa_metrics_uim_1*,ao_itoa_inventory_uim_ci_1*,ao_itoa_metrics_custom_1*
ALARM_CS_ID
Indicates the CS Id field used in cluster alarms
Default Value:
cs_id
ALARM_INDEX_ES
List of Alarm Indexes used to pull the data for the alarm clustering job     
Default Value:
ao_itoa_alarms_custom_1*,ao_itoa_alarms_uim_1*,ao_itoa_alarms_spectrum_1*,ao_itoa_alarms_anomaly_1* 
ALARM_OUT_INDEX
Alarm Index used to push the data processed by the alarm clustering job.     
Default Value:
ao_itoa_clusters_alarm_1*
ALARM_OUT_DOCTYPE_VER
Alarm Index outdoc type version used to push the data processed by the alarm clustering job.              
Default Value:
ENABLE_JAF
Flag to enable/disable CA Jarvis Analytics Framework in the alarm clustering job.        
Default Value: true
START_TIME
The time from which we take alarms for clustering.
Default Value:
6 months
STOP_WORDS
The list of words that are omitted when clustering happens.
JAF_ALARMJOB_MICRO_BATCH_INT
The micro batch interval (in mins) used to poll data from the topic.          
Default Value:
300            
ENABLE_ALARMCLUSTER_JOB
Flag to enable or disable clustering job. True indicates enable and false indicates disable.
Default Value:
true
ENABLE_PI_JAF_JOB
Flag to enable/disable the predictive insights job.
Default Value
: true
KAFKA_TOPICS
The Kafka topic from which the alarms have listened and processed for situation 
Default Value
: CAA_DTR__SV__ao_itoa_alarms_all_1_p1
IS_AC_JOB_UPDATED
Enable this flag to update alarm clustering environment variable
Default Value
:false
IS_PI_JOB_UPDATED
Enable this flag to update predictive Insights environment variable
Default Value
:false
PI_OUT_INDEX
Indicates the output doctype to store predictions.
Default Value
: itoa_alarms_prediction
PI_OUT_DOCTYPE_VER
Indicates the output doctype version to store predictions.
Default Value:
1
PI_JOB_INTERVAL
Indicates how often batch job runs, in hours.
Default Value:
1
PI_BASELINE
Indicates the baseline for calculations, in days.
Default Value:
42
ENABLE_AC_JAF_JOB
Enable/Disable Situation Alarms Batch Job.
Default Value
:true
AC_EXECUTOR_CORE
Number of cores allocated to Executor for Alarm Clustering Job
Default Value
:2
AC_EXECUTOR_INSTANCES
Number of instances for Executor for Alarm Clustering Job
Default Value
:2
AC_EXECUTOR_MEMORY
Memory allocated to each Executor for Alarm Clustering Job
Default Value
:4g
AC_DRIVER_MEMORY
Memory allocated to Driver for Alarm Clustering Job
Default Value
:6g
AC_DRIVER_CORE
Number of cores allocated to Driver for Alarm Clustering Job
Default Value
:1
MAX_ALARMS_FOR_ALARMCLUSTER 
The maximum number of alarms that can be clustered.
Default Value:
100000