ems Probe Deployment and Configuration

This article is for probe versions 8.4 or later
uimpga-ga
This article is for probe versions 8.4 or later
ems_AC
This article describes the deployment and initial configuration of the ems probe. After ems probe configuration is complete, you can configure event or alarm rules using the ems rule catalogs.
Contents
Overview
Before ems deployment, alarm messages flow through the CA UIM environment in the following manner:
  1. Probes and the baseline_engine send alarm messages with the subject "alarm".
  2. The alarm_enrichment probe receives the alarm messages, enriches them with user-defined data, then changes the alarm message subject to "alarm2".
  3. The nas probe receives the alarm2 message and performs further alarm processing. 
pre-ems nas flow
pre-ems nas flow
During the process of ems deployment and configuration:
  • The Alarm Routing Service is installed as a service within the trellis probe. The Alarm Routing Service performs the following tasks:
    • Selectively routes alarm messages to either ems or nas.
    • Duplicates any alarm messages that are sent to both ems and nas.
  • nas routing is modified. The alarm_enrichment probe is reconfigured to listen for the alarm message subject "alarm1". The 
    enrichment_subject
     key in the nas configuration file (nas.cfg) controls this behavior. By default, the nas probe continues to receive all legacy alarm messages until alarm routing is configured.
    Probes that send ems events natively do not require conversion or alarm routing.
  • ems routing is established. Once ems is deployed and routing rules are created, legacy alarm messages are sent to the Legacy Alarm Manager. The alarm messages are then converted into ems events.
ems routing overview
ems routing overview
If you decide to uninstall the ems probe, these changes are not automatically reverted. See the section Uninstalling or Reinstalling the ems Probe for more information about the correct procedure for uninstall or reinstall.
Within the ems probe, the nas legacy alarm messages that are sent from the Alarm Routing Service are converted to ems events. This conversion is configured by using the Event Type Conversion catalog.
flow within ems
flow within ems
CA UIM Requirements
You can only deploy the ems probe to your primary hub. Deployment to secondary or remote hubs is not supported. In CA UIM 8.5.1 or later, the ems probe is automatically deployed to the primary hub. 
(ems 8.50 or Earlier) Deploy the ems Probe
As of the CA UIM 8.5.1 release, ems is a required UIM component that is responsible for sending nas alarms to USM. Because of this change, ems is deployed automatically and you can skip this step.
If you are deploying ems into an environment that handles more than ten alarms per second steady state, deactivate nas before deploying ems. If you want to use both nas and ems in your environment, you can reactivate nas after configuring the alarm_routing_service.
The ems probe is available in the Admin Console Archive after installing or upgrading to CA UIM. Deploy the ems probe to the primary hub using Admin Console.
Deploy the Alarm Routing Service
The Alarm Routing Service is responsible for routing legacy nas alarms to ems. This service is contained within the trellis probe and is deployed independently from ems.
Follow these steps
  1. Deploy the alarm_routing_service package to the Primary Hub.
  2. Restart the trellis probe.
    After deployment, you can also use the
    alarm_routing_service_restart
    probe utility command in trellis to restart the Alarm Routing Service. The probe utility command does not force a restart of the trellis probe or any other trellis services.  
The alarm routing service is now active and you can configure ems routing rules.
Configure Legacy Alarm Routing
By default, all legacy alarms are sent to the nas probe. Configure the Alarm Routing Service to route these legacy alarms to ems.
Changes to the alarm routing service configuration file require a restart of the alarm_routing_service. You can do this by either:
  • Restarting the trellis probe.
  • Using the
    alarm_routing_service_restart
    probe utility command in trellis
Alarm Routing and USM
When you configure alarm routing, note the following behavior changes in USM:
  • Any alarms being routed to ems that were previously processed by nas still appear in USM with their old suppression counts. The suppression counts no longer increase, and the alarms are not auto-cleared.
  • Alarms that are routed to ems for the first time appear as new alarms in USM. These new alarms have a suppression count of 0.
  • Suppression counts are not carried over from the nas probe to the ems probe.
This behavior means that an alarm can appear in USM twice:
  • Once when nas initially processed the alarm.
  • Again when the alarm is routed to ems.
To remove the old alarms that nas initially processed, clear them manually in USM. Otherwise, the alarms remain in the Alarm View. You can add the
Route
column in the USM Alarm View to determine the originating alarm service for each alarm. 
Follow these steps
  1. Navigate to the Alarm View in USM.
  2. Click on the
    Table View
    button.
  3. Click 
    Actions
    Add Column, Route.
The
Route
column contains the originating alarm service for the alarm:
  • ems-routed alarm - The ems probe address
  • nas-routed alarm - The nas probe address
The Alarm Routing Service Configuration File
The Alarm Routing Service configuration file is located at
<UIM_Installation>\probes\service\trellis\service\alarm-routing-service\config\alarm-routing-service.cfg.
You can use the Alarm Routing Service configuration file to perform the following tasks:
  • Change the default routing destination from nas to ems.
  • Add routing rules to determine whether alarm messages are sent to ems, nas, or both.
Change the Default Alarm Destination
 All alarm messages are sent to nas if:
  • You do not create any routing rules.
  • All the specified routing rules fail.
You can alter the Alarm Routing Service configuration file to change this behavior.
This procedure assumes that the following statements are true:
  • You are changing the default alarm destination from nas to ems.
  • nas and ems are using the default settings for hub queue names and message subjects.
If this use case does not apply to you,
see the section
Change or Add Default Destinations
.
Follow these steps:
  1. Open the Alarm Routing Service configuration file.
  2. Locate the
    default_destination
    key within the
    <setup> 
    section and change
    nas_destination
    to
    ems_destination.
Change or Add Default Destinations
This section is only for users that have manipulated the alarm queue names in their CA UIM environment. Do not change the default destinations under any other circumstances.
The available destinations are located in the
<destinations>
section of the Alarm Routing Service Configuration file. By default, ems and nas are available as destinations using their default settings. If you have changed: 
  • The names of the hub queues for your alarm messages
  • The message subject that nas or ems listens for.
Then you can change one of the default destinations, or you can add a new destination.
Default Destination Guidelines
Before changing the default destinations in the Alarm Routing Service, note the following guidelines:
  • The Alarm Routing Service is only for use with probes that can configure their inbound subject names. In a typical CA UIM environment, the only probes that can configure their inbound subject names are:
    • ems
    • nas
    • alarm_enrichment
  • The
    field
    parameter in the
    <probe> 
    section is used by the receiving probe (ems or nas) to configure the subject that is specified in the 
    subject 
    field.
  • The Alarm Routing Service does not verify the value in the
    field
    parameter. If the
    field
    parameter is set to an incorrect value, then the Alarm Routing Service creates a queue with the incorrect name.
  • When the
    field
    parameter is changed, the alarm queue with the previously used name is removed.
    When the old alarm queue is removed, any alarm messages that are still in the queue are lost.
Destination Structure
The structure for a destination within the
<destinations>
section is as follows:
<destination_name> subject = legacy_alarm | alarm1 <probe> name = ems | nas section = setup field = legacy_alarm_manager_subject | enrichment_subject </probe> <hub> probe = ems | alarm_enrichment queue = legacy_alarm_manager_subject | enrichment_subject </hub> </destination_name>
<destination_name>
The destination name is used to indicate destinations within the Alarm Routing Service configuration file. You can use the destination_name as the key value in the default_destination and as the destination in any routing rules. The available pre-defined destinations are
nas_destination
and
ems_destination
.
subject
The message subject in the sent alarm message. This value is either 
legacy_alarm 
for ems or 
alarm1
for nas.
<probe>
The probe section contains information about the fields that provide alarm message data in either ems or nas configuration files. The setup section contains the following parameters:
  • name
    - The name of the probe receiving the alarm message. This value is either
    ems
    or
    nas
    .
  • section
    - The name of the section in the probe configuration file where the routing parameters are located. This value is
    setup 
    for both ems and nas.
  • field
    - The name of the field in the probe configuration file that controls the routing of alarm messages. This value is
    legacy_alarm_manager_subject
    for ems or
    enrichment_subject
    in nas.
<hub>
The hub section contains information about the hub queues that are used to transmit alarm messages across the UIM Message Bus. The hub section contains the following parameters:
  • probe
    - The probe that receives alarm messages. By default, this value is either
    ems
    or
    alarm_enrichment
    .
  • queue
    - The name of the hub queue that is used for alarm messages. By default, this value is either
    legacy_alarm_manager
    for ems or
    alarm_enrichment
    for nas
    .
Add Routes
Once you have determined the default destination for your alarm messages, you can create individual routes for your rules. Each route contains a set of filter criteria that determines whether alarms are sent to nas, ems, or both. If you do not add any routes or all the routes you create fail, then alarm messages use the default_destination located in the
<setup>
section of the Alarm Routing Service configuration file.
Routing Format
The ems routing rules are located in the
<routes>
section of the Alarm Routing Service configuration file. The format of each routing rule is as follows:
<routes> <routeName> order = 'Order Priority' <filters> 'Filter Parameters' </filters> destinations = alarm_destination </routeName> </routes>
<routes>
The <routes> section is a container for your user-defined routes. Routes are sets of filter criteria that determine where each alarm message is processed.
<routeName>
The <routeName> section contains the filter criteria for a route. The name of this element is a user-defined value. If an order is not specified, routes are run alphabetically using the designated <routeName>.
(Optional) Order
If an order is defined, routes are evaluated in the specified order, lowest value to highest, until a match is found. If two routes have the same value for order and both are matched, the routes are run alphabetically using the defined <RouteName>. Any numeric value is supported.
<filters>
The <filters> section contains the user-defined filter criteria. The following sections in this article describe the supported filter values. 
Destinations
The destinations element determines where alarm messages that match the filter criteria are sent. By default, the supported values are
ems_destination
or
nas_destination
. You can set the destination to
ems_destination, nas_destination 
to send alarm messages to both ems and nas. The destination name is case-sensitive. You can also use any custom destinations that you created in the
<destinations>
section.
If both ems_destination and nas_destination are specified in a route, alarm messages are copied and sent to both ems and nas. The ems and nas probes then process the messages according to their configured rules. This behavior can be useful for testing rule behavior in ems, but results in duplicated entries in USM and other parts of UMP.
Guidelines for Filters
Use the following guidelines when you create alarm filters:
  • You can create filters using any string field in an alarm message. However, we recommend that you limit your filters to the following fields:
    • Origin
    • Prid (probe name)
    • Robot
    • Source
    • Domain
  • Unless the order parameter is specified, rules are evaluated in alphabetical order using the defined
    <routeName>
    .
  • Multiple comma-separated values are supported.
    The
    *
    symbol indicates
    ALL
    in a filter. You cannot use the * symbol with other filter criteria on the same line. For example, a filter that contains:
    prid = cdm, *
    is not supported. Also, using the * symbol with any other filter value is redundant. For example, the following filters:
    Filter 1
    prid = * source = 10.20.30.100, 10.20.30.110
    Filter 2
    source = 10.20.30.100, 10.20.30.110
    return the same result.
  • OR
    logic is used within a line across comma-separated values.
  • AND
    logic is used across each property (between lines).
  • If a filter contains a blank value (for example,
    prid =
    ) the filter is ignored.
  • Regular expressions are not supported.
Filter Examples
This section contains filter examples that you can use.
Example - Filter with a Single Field
<filters> prid = net_connect </filters> destinations = ems_destination
Result:
The ems probe receives alarm messages from all net_connect probes in your UIM environment.
Example - Filter with Multiple Fields
<filters> source = 10.20.30.100 prid = net_connect </filters> destinations = ems_destination
Result:
The ems probe only receives alarm messages from net_connect probes that originate from the robot with the IP address 10.20.30.100.
Example - Filter with Multiple Values in Each Field 
<filters> source = 10.20.30.100, 10.20.30.110 prid = net_connect, cdm </filters> destinations = nas_destination
Result:
The nas probe receives alarms from
either
net_connect or cdm probes that originate from robots with
either
a 10.20.30.100 or 10.20.30.110 IP address.
Example - Filter on All
<filters> source = * </filters> destinations = ems_destination
Result:
The ems probe receives alarms that originate from any source.
Example - Multiple Routes Defined
<routes> <myGeneralCdmRoute> order = 20 <filters> prid = cdm </filters> destinations = nas_destination, ems_destination </myGeneralCdmRoute> <mySpecificCdmRoute> order = 10 <filters> prid = cdm source = 192.168.15.1 </filters> destinations = ems_destination </mySpecificCdmRoute> <myRedundantRoute> order = 30 <filters> prid = cdm source = * </filters> destinations = nas_destination </myRedundantRoute> </routes>
Result:
  1. Because of the defined relative order, the routes are evaluated in the following order:
    1. mySpecificCdmRoute
    2. myGeneralCdmRoute
    3. myRedundantRoute
      Since the myGeneralCdmRoute contains a filter that applies to all instances of the cdm probe, the myRedundantRoute rule is not reached or evaluated.
  2. Alarm messages from the cdm probe that originate from 192.168.15.1 are routed to ems.
  3. All other cdm probe alarm messages are routed to both ems and nas.
(nas Replication Only) Create Hub Queues to Route Alarms to the Primary Hub
Currently, only alarms that arrive at the Primary Hub are processed through the Alarm Routing Service. If you are using nas replication in your environment, you can create queues that:
  • Listen to the subject alarm.
  • Route any legacy alarm messages to the Primary Hub.
Once the legacy alarm messages are routed to the Primary Hub, they are processed by the Alarm Routing Service and ems.
Use the Hub AC Configuration topic and the following Alarm Routing Service-specific guidelines to create your queues:
  • Each Secondary Hub requires a queue that sends alarms to the Primary Hub for routing to ems.
  • During queue creation, set the
    Subject
    field to
    alarm
    .
  • Both
    attach
    and
    post
    queue types are supported.
These extra queues do not alter the alarm_enrichment or nas queues, nor do they affect the configured nas replication. All alarms that are generated on a Secondary Hub still pass through the alarm_enrichment and nas probes that are on that Hub.
(Optional) Deactivate nas and alarm_enrichment
If you want to use ems exclusively, you can deactivate the nas and alarm_enrichment probes.
Follow these steps:
  1. Ensure that all alarms are being routed to ems. This action prevents alarms from being stuck in the alarm1 queue.
    1. Verify that the
      <setup>
      ,
      <hub>
      , and
      <routes>
      sections in the Alarm Routing Service configuration file are configured correctly for your UIM environment.
  2. Deactivate and remove the nas probe using Admin Console or Infrastructure Manager.
If you decide to reinstall the nas probe after uninstalling it:
  • The Alarm Routing Service is not uninstalled, nor are any configuration settings changed.
  • The alarm routing is not changed. Change the
    enrichment_subject
    in nas back to "alarm1" manually using the process that is described in Uninstalling or Reinstalling the ems Probe.
(Optional) Change the Settings in the Admin Console GUI
Extra configuration options are available in the ems probe GUI in Admin Console. These settings can be altered any time after initial probe deployment.
Change the Log File Parameters
You can change both the ems log file size and the level of the alerts that are documented. The default location of the ems log file is
<UIM Install>\probes\service\ems.
Follow these steps:
  1. Navigate to the top-level ems node in the Admin Console probe GUI.
  2. Change the
    Log level
    field desired logging level. The log levels that you can specify are:
    • 0 - Fatal
      - Severe messages are logged.
    • 1 - Error
      - Errors are logged.
    • 2 - Warn
      - Warnings are logged.
    • 3 - Info
      - Informational messages are logged.
    • 4 - Debug
      - Debugging messages are logged.
    • 5 - Trace
      - Tracing/low-level debugging messages are logged.
  3. Change the
    Maximum Log File Size(KB)
    field to the desired log size in KB.
Change the Data Retention Period
By default, transaction history for the ems probe is stored for 30 days. You can change the retention settings so that the ems .csv files are purged on a more or less frequent basis.
Follow these steps:
  1. Navigate to the top-level ems node in the Admin Console probe GUI.
  2. Change the
    Transaction Log Retention (Days)
    to the desired number of days.
Enable NAS-Style Lifecycle Messages
In later versions of the ems probe (8.50 or later), this option is activated by default.
Some CA UIM components depend on alarm lifecycle messages that are sent from the nas probe. Some examples include:
  • Gateway probes
  • Integrations with service desk software such as CA Service Desk
  • Any component that uses the following alarm message types:
    • alarm_assign
    • alarm_close
    • alarm_new
    • alarm_update
If you would like to use ems with these components, you can enable lifecycle messages in ems. If you do not require alarm lifecycle messages, leaving this option off provides a small reduction in bus message traffic.
Follow these steps:
  1. Navigate to the top-level ems node in the Admin Console probe GUI.
  2. Select
    Send NAS-style Lifecycle Messages. 
  3. Restart the ems probe.
(ems 8.47 or earlier) Note:
After you enable NAS-style lifecycle messages, alarm messages appear in the Infrastructure Manager Alarm Console. These alarm messages contain incorrect dates in the
Time Received
,
Time Origin
, and
Time Arrival
fields. You can safely ignore these messages, as they have no impact on ems and do not appear in USM.
(ems 9.00 or Later) Activate the Export of Open Alarms to the UIM Database
When this option is active, open alarm data is exported to the ems_alarm table in the UIM database. This option also updates tables that store URL actions and custom property data other than custom properties 1 through 5. Open alarm data from both ems and nas is also exported to a view named 
uim_alarm_view
.
Follow these steps:
  1. Navigate to the top-level ems node in the Admin Console probe GUI.
  2. Select
    Enable UIM DB Alarm Sync. 
  3. Restart the ems probe.
Deactivating open alarm export after activating the option does not remove the associated tables or data from the UIM database. 
(ems 9.00 or Later) Change the Open Alarm Sync Interval
By default, open alarms are exported to the UIM database every 90 seconds. 
Follow these steps:
  1. Navigate to the top-level ems node in the Admin Console probe GUI.
  2. Change
    UIM DB Alarm Sync Interval (Seconds) 
    to the desired value. 
     
  3. Restart the ems probe.
(Optional) Change the Settings in the Raw Configure Menu
More settings are available to advanced users through the Raw Configure menu in either Admin Console or Infrastructure Manager.
Deactivate Host Name Resolution
By default, host name resolution is turned on and the IP address of the event message originator resolves to a host name. You can turn off host name resolution if:
  • You do not want to use system resources to perform host name resolution.
  • You want to retain an IP address in the source field of the message.
Follow these steps:
  1. Navigate to the
    setup
    folder in the ems Raw Configure menu.
  2. Change the key
    disable_hostname_resolution
    to true.
Once the change is made, ems retains the IP address in the source field of the message.
Deactivate Close on Auto Clear
By default, close on auto clear is turned on. If you do not want to close alarm messages when they are in a clear state for debugging purposes, you can turn off this option.
Unless directed by CA support, we recommend leaving close on auto clear on. Turing off close on auto clear can result in the accumulation of unclosed alarms.
Follow these steps:
  1. Navigate to the
    setup
    folder in the ems Raw Configure menu.
  2. Add the follow value-key pair:
    • close_on_auto_clear = false
(ems 8.47 or Earlier) Uninstall or Reinstall the ems Probe
As of CA UIM 8.5.1, ems is a required component. Only use this procedure if you are using a previous version of CA UIM.
If you remove the ems probe from your CA UIM environment:
  • The Alarm Routing Service is not removed.
  • The
    enrichment_subject
    in the nas configuration file (
    nas.cfg
    ) remains "alarm1".
Before you deactivate and remove the ems probe, use the following procedure to ensure that no alarms are lost.
Follow these steps:
This procedure assumes that legacy alarm messages are being sent to the ems probe. If you did not:
  • Change the default routing destination to ems.
  • Configure any routing rules to send alarms to ems.
Skip to step 3.
  1. In the Alarm Routing Service configuration file, change the default destination back to nas and remove any routing rules. This action prevents alarms from being stuck in the legacy_alarm queue.
  2. Enter the Raw Configure menu for the nas probe.
  3. Navigate to the
    setup
    folder. 
  4. Change the
    enrichment_subject
    key from "alarm1" to "alarm".
  5. If you had deactivated the nas probe, deploy and reactivate it.
  6. Deactivate and remove the ems probe. 
If you reinstall the ems probe after an uninstall:
  • Do not reinstall the Alarm Routing Service.
  • Change the
    enrichment_subject
    back to "alarm1" manually using the process that is described in this section.
  • Reconfigure any ems routing rules.
After you have configured the ems probe, you can begin writing event and alarm rules.