Create Notifications for Alerts

apmsaas
HID_Notifications
DX APM
lets you create notifications for alerts. Notifications can automatically relay alerts from Application Performance Management through internal services, such as distribution lists, to external services through the REST API, or to PagerDuty, an incident resolution platform.
For more information about setting up email notifications, see Configure Email Notifications for Alerts.
PagerDuty
PagerDuty performs incident analysis and sends notifications directly to designated IT administrators for resolution. Use the PagerDuty integration to monitor your system for critical events through a defined path, respond quickly, and keep your system operating with minimal downtime. For more information about PagerDuty capabilities, go to https://www.pagerduty.com.
To work efficiently in the two systems, follow these rules:
  • An alert in
    DX APM
    triggers an incident in PagerDuty. PagerDuty then transmits the incident as a ticket to a designated recipient.
  • An alert that is cleared in
    DX APM
    resolves the incident in PagerDuty. Resolving an incident in PagerDuty does not clear the alert in Application Performance Management.
Incident Workflow
When an alert is generated and matches the service configuration settings,
DX APM
automatically triggers an incident. Incidents in PagerDuty contain the following information for an alert to identify the affected component:
  • alertStartTime
  • alertState
  • alertName
  • alertId
  • vertexId
  • vertexAttributes
    (for example, type, name
    )
PagerDuty then handles incidents according to configured policies and user settings.
A cleared alert in
DX APM
automatically resolves an incident in PagerDuty. The notification delay is a length of time that
DX APM
waits before triggering an incident in PagerDuty. The notification delay is useful for reducing incidents that are triggered by transitory alerts.
Create and Modify a PagerDuty Notification
You must have a PagerDuty account to create a PagerDuty Notification. The PagerDuty account lets you configure notification behavior and receive a unique API User Token and Integration Key for use with
DX APM
.
Follow these steps:
  1. Log in to PagerDuty or create an account.
  2. Copy the API User Token and the Integration Key from PagerDuty.
  3. In
    DX APM
    , click
    Notifications
    .
  4. Click
    Create a PagerDuty Notification
    .
  5. Create a Notification Name.
  6. Paste the API User Token and Integration Key from PagerDuty.
  7. Select the severity for the notification.
  8. Click
    Save
    .
  9. (Optional) Select the notification that you want to change and click
    Edit
    .
  10. (Optional) Select the notification that you want to delete and click
    Delete
    .
You created and modified a PagerDuty Notification as required.
REST API Notifications
DX APM
lets you receive notifications about alert changes through the REST API. The REST API is the pulling interface that allows integrations to run behind corporate firewalls. Use the API to integrate notifications with third-party components, such as various dashboards. The following diagram shows the REST API within the Application Performance Management infrastructure:
REST API Notification
REST API Notification
Create and Modify a REST API Notification
Follow these steps:
  1. In
    DX APM
    , click
    Notifications
    .
  2. Click
    Create a REST API Notification
    .
  3. Create a Notification Name.
  4. Create a unique proxy token or click
    Generate Token
    .
  5. Click
    Save
    .
  6. (Optional) Select the notification that you want to change and click
    Edit
    .
  7. (Optional) Select the notification that you want to delete and click
    Delete
    .
You created and modified a REST API Notification as required.
Integrate REST API Notifications
Access the REST API to integrate REST API Notifications with third-party components.
Follow these steps:
  1. Generate a security token in the
    DX APM
    Settings
    . Save the security token somewhere for later use. For more information about how to generate a security token, see API Authentication and Authorization.
  2. Navigate to
    Notifications
    and copy the proxy token from the REST API Notification that you want to integrate. Save the proxy token somewhere for later use
    You can use the same proxy token for several notifications in
    DX APM
    . The REST API returns the alert notifications from all these notification configurations. The REST API call labels the proxy token as
    proxyKey
    . The
    proxyKey
    identifies the REST API client.
  3. Set the last version value to 0 to receive non-OK statuses (Caution or Danger) for all active alerts. For future REST API calls, use the last version value that was returned in the previous call to avoid receiving unchanged statuses.
  4. Copy the host number from your
    DX APM
    URL. Save the host number somewhere for later use.
  5. Paste the host number, security token, proxy token, and last version into the following POST command to call the REST API:
POST
Host: https://<YOUR HOST NUMBER>.apm.cloud.ca.com/apm/appmap/private/graph/recentstatuschanges
Accept: application/json
Content-Type: application/json
Authorization: Bearer <YOUR SECURITY TOKEN>
{ "lastVersion" : "<LAST VERSION>", "proxyKey" : "<YOUR PROXY TOKEN>" }
Response:
The REST API sends a JSON object as the response. The response contains alert changes for all notifications with the same proxy token value that happened after the last call. The response also returns the version field for the following call.
In the response, each notification contains the following fields:
    • status
      – the severity of the alert (OK, CAUTION, DANGER)
    • alertName
      – name of the alert
    • time
      – occurrence of the status change in milliseconds since the Unix epoch
    • vertex
      – the alerted vertex including all the vertex attributes
Example of a Response:
{ "items": [ { "vertex": { "agent": [ "CA APM Demo Host|Tomcat|CA APM Demo Agent - Tomcat" ], "hostname": [ "ca apm demo host" ], "Source cluster": [ "Enterprise Team Center" ], "name": [ "Apps|ReportingEngine|URLs|Default" ], "agentDomain": [ "SuperDomain" ], "IsDemo": [ "Yes" ], "processedBy": [ "FrontendVertexIdentifier" ], "type": [ "GENERICFRONTEND" ], "applicationName": [ "ReportingEngine" ] }, "status": "DANGER", "time": 1507025865000, "alertName": "SuperDomain:SaaS:Frontend Errors" }, { "vertex": { "agent": [ "CA APM Demo Host|Tomcat|CA APM Demo Agent - Tomcat" ], "IsExperience": [ "Yes" ], "agentDomain": [ "SuperDomain" ], "IsDemo": [ "Yes" ], "type": [ "GENERICFRONTEND" ], "servletMethod": [ "service" ], "Experience": [ "Apps|ReportingService|URLs|Default on ca apm demo host (GENERICFRONTEND)" ], "hostname": [ "ca apm demo host" ], "Source cluster": [ "Enterprise Team Center" ], "name": [ "Apps|ReportingService|URLs|Default" ], "serviceId": [ "ApplicationService" ], "processedBy": [ "FrontendVertexIdentifier" ], "applicationName": [ "ReportingService" ] }, "status": "OK", "time": 1507025865000, "alertName": "SuperDomain:SaaS:Frontend Errors" } ], "version": 1507026459020 }
Authentication HTTP Status and Error Codes
If the authentication of the request fails, the resource server returns an HTTP error code and response header with details about the error.
  • 401 Unauthorized – your security token (sent in HTTP header) is not valid for the given URL.
  • 403 Forbidden – the proxy token is not matching any notification configuration.
For more information, see HTTP status code definitions.
ATC Notification Proxy
The ATC Notification Proxy is an implementation of the REST API client, which can run inside the Infrastructure Agent as an extension. This extension periodically calls the REST API to obtain all notifications. The extension then forwards these notifications one by one to other REST APIs (destinations), for example, OMNIbus. Each notification can be converted into various formats that are suitable for the destination REST API.
Configure the ATC Notification Proxy in the following two files:
  • atcnotificationproxy.properties
    – set the ATC URL, security token, proxy token, destination URL, and the pulling interval here.
    Example:
    # sleep between fetches in seconds
    atcnotification.sleep=10
    # url including protocal (http/https) to your ATC backend.
    atcnotification.atcBaseURL=https://123456.prod.apm.cloud.ca.com/
    # public API authentication token (can be obtain in ATC-> Security -> generate new token)
    # it is recommended to use, never expire option.
    atcnotification.securityToken=e45038c6-ffeb-45b6-ad74-4bfca43ec639
    # token defined in the notification. Can be set by user or generated. You can use same
    # token for multiple notifications in case you want your proxy to fetch and forward multiple
    # notifications
    atcnotification.proxyKey=myToken
    # url where the notification message will be forwarded
    atcnotification.omnibusURL=http://localhost:8080/
  • ssage.template
    – define the format of the notification message that goes to the destination REST API. The template allows you to use various placeholders and inject fields from Application Performance Management. All placeholders are listed in the template.
    Example:
    ## template to be send to Omnibus. Placeholders might be used:
    ## ${APM.alertName} - name of the alert
    ## ${APM.statusText} - status (DANGER, CAUTION, OK)
    ## ${APM.occurrence} - Alert occurrence in GMT format
    ## ${APM.Vertex.type} - value of the attribute (type) of the alerted component
    ## ${APM.Vertex.name} - value of the attribute (name) of the alerted component
    ## ${APM.Vertex.<anyattribute>} - value of the attribute of the alerted component
    {
    "source": "CA_APM",
    "CustomerCode": "IBM",
    "msg": "${APM.alertName} in ${APM.statusText} state",
    "InstanceId": "${APM.Vertex.applicationName}",
    "InstanceSituation": "${APM.alertName}",
    "origin": "${APM.Vertex.hostname}",
    "hostname": "${APM.Vertex.hostname}",
    "Component": "${APM.Vertex.name}",
    "ComponentType": "${APM.Vertex.type}",
    "Severity": "${APM.statusText}"
    }