Integrate With DX Operational Intelligence

2
casp1043
2
Overview
The integration between
DX NetOps Spectrum
and DX Operational Intelligence allows you to leverage data from
DX NetOps Spectrum
and helps to analyze, correlate, and proactively resolve network issues. The SpectrumDataPublisher is a utility/service in
DX NetOps Spectrum
that publishes the
DX NetOps Spectrum
data to the analytics platform.
This release supports sending Topology inventory and its relations from
DX NetOps Spectrum
to Topology Analytics Service (TAS) using
DX NetOps Spectrum
Data Publisher (Spub).
  • SpectrumDataPublisher synchronizes
    DX NetOps Spectrum
    inventory to the TAS database used by DX OI.
  • SpectrumDataPublisher pushes all the
    DX NetOps Spectrum
    devices and their relations among the adjacent devices to the TAS database.
  • SpectrumDataPublisher converts
    DX NetOps Spectrum
    devices into Vertices, and devices relations to Edges before sending them to TAS.
The following data is synchronized from
DX NetOps Spectrum
to DX Operational Intelligence:
  • Alarms
    Reconciliation does not happen when elastic has a large number of alarms. There is a limitation of 10000 alarms from DX OI.
  • Network Configuration Manager- Change events (Disabled by default, enable if required)
  • Topology
The following data are not supported in this release as the related data is pushed as part of Topology (TAS Integration):
  • Inventory
  • Groups (Global Collection and Container)
  • VNAInventory (Disabled by default, enable if required)
  • Metrics: Device count and Device availability (based on Model Type and Group)
What's New in 10.4.3.1!
From 10.4.3.1, you can use the DX OI Service Account information to get alarms from OI for alarm reconciliation. The service startup clears the stale alarms in DOI using the service account information. For more information, see Service Account username.
What's New in 10.4.3!
From the 10.4.3 release, the DX Operational Intelligence connector uses the consolidated ingestion endpoint to push all inventory/topology and alarms data. The 10.4.3.x release supports only DX Operational Intelligence SaaS.
Upgrade
DX NetOps Spectrum
Data Publisher
From the 10.3.1 release, there are considerable changes that are done to the
DX NetOps Spectrum
Data Publisher. It is recommended to upgrade the
DX NetOps Spectrum
Data Publisher version to verify that the
DX NetOps Spectrum
Data Publisher is running the same version of
DX NetOps Spectrum
.
During the DX Operational Intelligence Connector upgrade to the current release, the installer upgrades the necessary files with new parameters.
Follow these steps:
  1. Log in to the OneClick client.
  2. Navigate to the OneClick Administration page and select
    Analytics Configuration
    in the left panel.
    The
    Analytics Configuration
    page opens.
  3. Under the
    DX NetOps Spectrum
    - DX OI Connector
    section, select the
    DX NetOps Spectrum
    - DX OI Connector
    link to download the installer.
    The SpectrumDataPublisher.jar file is downloaded to the default Downloads folder on your computer.
  4. Double-click the JAR file.
    The
    DX NetOps Spectrum
    Data Publisher
    install wizard opens.
  5. Select 
    Next
    to install the folder location.
  6. Select
    Install
    .
    A pop-up window appears displaying the current version and the version you are upgrading to.
  7. Select
    OK
    to proceed.
    The
    DX NetOps Spectrum
    Data Publisher is upgraded.
  8. Select
    Done
    to close the installation wizard.
DX NetOps Spectrum
-DX OI Integration Architecture
The following diagram explains how the
DX NetOps Spectrum
-DX Operational Intelligence integration works:
Spectrum-DX Operational Intelligence
Compatibility Matrix
For more information about the integration of
DX NetOps Spectrum
with other CA products, see the following links:
Installing the SpectrumDataPublisher
The
DX NetOps Spectrum
OneClick page allows you to download and install the SpectrumDataPublisher JAR file. The
DX NetOps Spectrum
Data Publisher version should be the same as the version of the
DX NetOps Spectrum
.
Follow these steps:
  1. Log in to the OneClick WebApp.
  2. Navigate to the
    OneClick Administration
    page.
  3. Select the
    Analytics Configuration
    option from the panel on the left.
    The
    Analytics Configuration
    page opens in a separate window.
  4. Under the
    DX NetOps Spectrum
    - DX OI Connector
    section, select the
    DX NetOps Spectrum
    - DX OI Connector
    link to download the installer.
    The SpectrumDataPublisher.jar file is downloaded to the default Downloads folder on your computer.
  5. Perform one of the following steps:
    • In
      Windows:
      Double-click on the jar to open the install wizard and enter the
      Install Directory
      . By default, the path is:
      C:\win32app\
      Th
      e
      DX NetOps Spectrum
      Data Publisher Install
      wizard opens.
    • In
      Linux
      perform one of the following tasks to start the installation:
      • Execute the following command for silent installation:
        java -jar SpectrumDataPublisher.jar -i silent -DUSER_INSTALL_DIR="<install directory>"
        For example:
        java -jar SpectrumDataPublisher.jar -i silent -DUSER_INSTALL_DIR="/opt"
      • Execute the following command to export DISPLAY to a Windows Server and start GUI based installation:
        Export DISPLAY=<machine-name>:0 and then run the following command:
        java -jar SpectrumDataPublisher.jar
  6. Select
    Next
    .
  7. Select Install for the
    DX NetOps Spectrum
    Data Publisher
    to be installed.
  8. Select
    Done
    to close the Install wizard.
    After the installation is complete, the SpectrumDataPublisher folder is created in the install folder location.
Configuring the SpectrumDataPublisher
To synchronize data from
DX NetOps Spectrum
to DX Operational Intelligence, configure the SpectrumDataPublisher.
Follow these steps:
  1. Navigate to the
    DX NetOps Spectrum
    Data Publisher install location.
  2. Open the 'config' folder.
  3. Open the ConnectorConfig.xml file, and configure the following:
    1. SpectrumConfiguration
      section: The following code block shows a sample file content, check the self-explanatory comment against the parameter in the file.
      1. Enter the
        DX NetOps Spectrum
        OneClick host server details in this section.
        For example:
        <SpectrumConfiguration> <OneClickServerUrl></OneClickServerUrl> <!-- give OneClickServerUrl. example: http://spectrum-123.net:8080/ --> <ConfigFile>SpectrumConfig.xml</ConfigFile> <WebappLaunchUrl></WebappLaunchUrl> <!-- give webapp url. example: http://spectrum-123.net:8080/spectrum/webapp/ --> </SpectrumConfiguration>
        If you want to use the Secure Sockets Layer (SSL) protocol to encrypt communications between TAS and the SpectrumDataPublisher, you must import SSL (https) Certificate to TAS endpoint.
        Launch OneClick Console or WebApp from DX OI Alarms
        When the
        WebappLaunchUrl
        property is empty OneClick console is launched and when you provide the WebApp URL, OneClick WebApp is launched.
    2. Destinations
      section:
      The following table displays the required parameters and explains how to obtain their values in different DX Operational Intelligence environments:
      The Connector Parameters section in the DX SAAS Settings page provides all the mandatory parameters that are required to configure SpectrumDataPublisher. For more information, see Connector Parameters.
      1. DestinationDefaultTenant:
        Parameter Name
        Description
        DX OI On-Prem 21.X
        DX OI SaaS
        DestinationDefaultTenant
        Specifies the tenant ID or Cohort ID.
        1. Obtain the Elastic endpoint. Depending on your environment, run the appropriate command.
          • Openshift:
            oc get routes -n<your-namespace> | egrep jarvis-es
          • Kubernetes:
            kubectl get ingress -n<your-namespace> | egrep jarvis-es
        2. Run a query against the elastic endpoint to list all tenants.
          http(s)://<elastic-endpoint>/ao_dxi_tenants_1_1/_search?size=200&pretty
        3. Locate your tenant name and extract the value of the associated "tenant_id" parameter.
        1. Navigate to the Dx SaaS, Settings,  Connector Parameter page
        2. Note down the tenat_id which is equal to the
          Cohort ID
      2. DestinationUrl:
        Parameter Name
        Description
        DX OI On-Prem 21.X
        DX OI SaaS
        DestinationUrl
        Specifies gateway endpoint.
        • Openshift:
          oc get routes -n<your-namespace> | egrep gateway
        • Kubernetes:
          kubectl get ingress -n<your-namespace> | egrep gateway
        1. Navigate to the Dx SaaS, Settings, Connector Parameter page
        2. Note down the
          TAS endpoint
      3. Tas:
        Parameter Name
        Description
        Tas
        Set the parameter to
        true
        to enable TAS inventory synchronization.
      4. AlarmReconcileConfiguration
        : Use the AlarmReconcileConfiguration parameter to clear the stale alarms from the DX OI. Modify the AlarmReconcileConfiguration parameters as shown in the following table:
        Parameter Name
        Description
        DX OI On-Prem 21.X
        DX OI SaaS
        Enable
        Set the parameter to
        true
        to clear the stale alarms during service startup.
        OIUrl
        Specifies the DX Operational Intelligence host URL.
        • Openshift:
          oc get routes -n<your-namespace> | egrep dxi-adminui
        • Kubernetes:
          kubectl get ingress -n<your-namespace> | egrep dxi-adminui
        • SaaS NA= https://axa.dxi-na1.saas.broadcom.com
        • Saas EU= https://axa.dxi-eu1.saas.broadcom.com
        BearerToken
        Specifies the bearer or also known as DX Auth token.
        See the Get DX AuthToken Used in Alarm Reconcile Configuration section.
        Applicable only in the 10.4.3 release. This parameter does not exist in the 10.4.3.1 and later releases. The service startup clears the stale alarms in DOI using the service account information.
        Reconciliation does not happen when elastic has a large number of alarms. There is a limitation of 10000 alarms from DX OI.
    3. TasConfiguration
      section: Set the value of the parameters as per your requirements.
      For example, FullSyncInterval - Specifies the interval between two full synchronizations in hours. By default, full synchronization is set to 720 hours. To synchronize the TAS inventory more frequently, you can, for example, set it to 24 hours.
    4. Save and close the file.
After the configuration is complete, start the SpectrumDataPublisher service.
Start the SpectrumDataPublisher Service
To start the SpectrumDataPublisher service, follow these steps:
  1. Perform one of the following steps to start the SpectrumDataPublisher service:
    • In Windows, from the command line, run the
      run.bat start
      command.
    • In Unix, from the console, run the
      run.sh start
      command.
  2. When you run the 'run.bat start' or 'run.sh', the script prompts you to enter the following login details.
    1. Enter OneClick username:
      DX NetOps Spectrum
      username.
    2. Enter OneClick password:
      DX NetOps Spectrum
      user password.
    3. (Only for v10.4.3.1 and later)
      Enter DOI ServiceAccount username -- if applicable:
      Enter the username of the Admin user of the Tenant as configured in the DX OI server.
      • To enable communication between the On-Premise systems and DX SaaS using DX Gateway in the SAML enabled environments, create a Service Account User, and configure this user in DX Gateway. The DX OI Service Account information lets you reconcile alarms between
        DX NetOps Spectrum
        and DX OI. The service startup clears the stale alarms in DOI using the service account information.
      • Service account user is available only for DX OI SaaS and for the On-Premise 20.2 and later versions. If you are upgrading from previous versions, users have to reset the connection to provide service account details for the alarm reconciliation.
      • For more information on how to create the Service Account, see the Create a Service Account User section in the DX SAAS documentation.
    4. (Only for v10.4.3.1 and later)
      Enter DOI ServiceAccount password -- if applicable:
      Enter the password of the Admin user of the Tenant as configured in the DX OI server.
      DOI service account credentials are required when you intend to reconcile alarms from DX OI.
    5. Enter the TAS Bearer Token
      : The following table shows how to get this token:
      Parameter Name
      Description
      DX OI On-Prem 21.X
      DX OI SaaS
      TAS Bearer Token
      Specifies TAS Bearer Token.
      1. Generate a Tenant or TAS Bearer token from APM. For more information, see Generate Security Token page in APM documentation.
      2. If APM is not available, generate a Tenant token from Cluster Management using the master admin account.
      1. Navigate to the Dx SaaS, Settings, Connector Parameter page.
      2. Generate a new Ingestion Token.
    6. Enter username for destination proxy -- if applicable:
      Enter a proxy server user name.
    7. Enter password for destination proxy -- if applicable:
      Enter the password of the proxy server.
    The script validates the information you provided and displays the success or failure messages. If the script successfully completes the validates, the SpectrumDataPublisher service starts.
SpectrumDataPublisher Validation at Service Start
The SpectrumDataPublisher validates the user inputs and all the connections at the service startup using the following criteria before starting the service:
  • <SpectrumConfiguration> URL is empty.
  • <DestinationUrl> is empty.
  • It runs the test connection validations to all given URLs, and checks for the following errors:
    • SSL Error - This error comes when https configured, and certificates not imported.
    • HTTP 401 - if the authentication fails.
    • HTTP 407 - if proxy details are incorrect.
    • UnknownHost exception - if the provided URL is not valid.
    You can check the
    SpectrumDataPublisher\logs\SpectrumDataPublisher.log
    file for the exact error description.
SpectrumDataPublisher service does not start if any one of the validations fails.
Post-installation
To avoid manual restart of the SpectrumDataPublisher service, change the 'Startup Type' from 'Manual' to 'Automatic' in Windows Services.
Stop the SpectrumDataPublisher service
If you want to stop the SpectrumDataPublisher-service, run the following command:
Windows:
run.bat stop
Linux:
run.sh stop
Restart the SpectrumDataPublisher service
If you want to restart the SpectrumDataPublisher-service, run the following command:
Windows:
run.bat restart
Linux:
run.sh restart
On Windows machines, the SpectrumDataPublisher-service can be stopped and restarted through the Windows Services Console.
Import SSL (https) Certificate into SpectrumDataPublisher
If you want to use the encrypted communication (https protocol) between
DX NetOps Spectrum
DataPublisher and integrated products, you must import the SSL/https certificate from the respective products into the SpectrumDataPublisher.
Follow these steps to import the certificate:
  1. Ensure that you export and copy the 'https' certificate to the server where the SpectrumDataPublisher is installed.
  2. Perform the following steps to download the APM services gateway and
    DX NetOps Spectrum
    OneClick endpoints SSL/HTTPS certificate:
    • Run the following command:
      If DX APM service gateway (TAS) endpoint is using SSL/HTTPS protocol:
      openssl s_client -connect TAS_Endpoint:<port> < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > tas.cer
      If
      DX NetOps Spectrum
      OneClick is using SSL/HTTPS protocol:
      openssl s_client -connect Spectrum_OneClick_Hostname:<port> < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ocserver.cer
      OR
    • Access the TAS endpoint or also called APM gateway using HTTPS in a browser and save the certificate from the site information.
  3. Run the following command to import the certificate into the Keystore of SpectrumDataPublisher.
    If DX APM service gateway (TAS) endpoint is using SSL/HTTPS protocol:
    keytool -importcert -alias tas -file tas.cer -keystore <SpectrumDataPublisher-HOME>/Security/cacerts
    If DX NetOps spectrum OneClick is using SSL/HTTPS protocol:
    keytool -importcert -alias tomcatssl -file OCServer.cer -keystore <SpectrumDataPublisher-HOME>/Security/occerts
  4. When prompted, provide the Keystore 'changeit' as a password.
Disable SSL Certificate Validation
You can disable the verification to check the validity of certificates. You can disable SSL Certificate Validation in the following scenarios:
  • When you do not have the SSL certificates.
  • When you get an SSL exception, even after you import the SSL certificates.
Add the following tags in the
ConnectorConfig.xml
file:
<DisableSSLHostnameVerifier>true</DisableSSLHostnameVerifier> <TrustAllX509Certificates>true</TrustAllX509Certificates>
Reset the Login Details of SpectrumDataPublisher service
In case you forgot the login password for the SpectrumDataPublisher service, you can reset the same by using the following commands. Note that, when you reset the login details, the stored configuration details are deleted and you should provide the details again in the ConnectorConfig.xml file.
From 10.4.3, after creating the login password for SpectrumDataPublisher, it asks for the login details that must be encrypted/ decrypted with the password.
Follow these steps
  1. Open the command/shell prompt from the SpectrumDataPublisher installation folder.
  2. Run one of the following commands based on your OS:
    Windows:
    run.bat reset
    Linux:
    run.sh reset
    at the prompt enter, the requested information.
  3. Enter the login details as described in the Start the SpectrumDataPublisher Service section.
  4. Start the SpectrumDataPublisher using one of the following commands based on your OS:
    Windows:
    run.bat start
    Linux:
    run.sh start
Logging Configuration for SpectrumDataPublisher
You can find the following log files in the SpectrumDataPublisher/logs/ folder:
  • The
    SpectrumDataPublisher.log
    shows the information and error messages of all the synchronizations.
  • The
    topology.log
    shows the inventory data sent from
    DX NetOps Spectrum
    to TAS.
  • The
    alarm.log
    shows the alarm data sent from
    DX NetOps Spectrum
    to Jarvis..
  • The
    ncm.log
    shows the NCM data sent from
    DX NetOps Spectrum
    to Jarvis.
Note the following points about the logs:
  • To set the
    SpectrumDataPublisher.log
    level change the value of
    rootLogger.level
    to info, debug, error, or fatal in the log4j2.properties file.
  • Set the
    logger.topoLogger.level
    parameter to
    trace
    in log4j2.properties to log the topology data.
    logger.topoLogger.level=trace
  • Set the
    logger.alarmLogger.level
    parameter to
    trace
    in log4j2.properties to log the alarm data.
    logger.alarmLogger.level=trace
  • Set the
    logger.ncmLogger.level
    parameter to
    trace
    in log4j2.properties to log the NCM data.
    logger.ncmLogger.level=trace
  • The maximum size of the log file is 100 MB. When the file reaches the maximum size, a backup file is created. The name of the backup file is SpectrumDataPublisher.log prefixed with a timestamp. The normal logging is continued in the SpectrumDataPublisher.log file.
(Only in 10.4.3) Get DX OI Agent Bearer Token Used in Alarm Reconcile Configuration
This section is applicable only for v10.4.3 and not applicable for 10.4.3.1 and later releases.
You need a DX OI agent bearer token when you configure the SpectrumDataPublisher. Note this bearer token for future use.
The token expires periodically, hence you must fetch the token afresh while starting the service.
Follow these steps:
  1. Refresh the DOI page after login.
  2. Press CTRL+SHIFT+R on the keyboard.
  3. Navigate to the Network.
  4. Refresh the page again.
  5. Click on any API in the
    Network
    tab.
  6. Scroll down, you can see the authorization: Bearer.
    Spectrum-DX Operational Intelligence
Filter Spectrum Alarms going to OI
This feature enables you to filter the alarms pushing to OI using the alarm filter option in Spectrum OneClick Client. You can create an alarm filter in Spectrum OneClick Client using multiple options.
Example:
With Severity-Critical, you can see only critical alarms, other severity alarms are filtered and does not appear.
Configuration setup
  1. Stop the SpectrumDataPublisher.
  2. Open Spectrum OneClick Client.
  3. Go to Alarms tab.
  4. Create an alarm filter in Spectrum OneClick Client in the Alarms tab.
  5. Use the created alarm filter name in SpectrumDataPublisher to synchronize the alarms to Jarvis.
  6. Configure SpectrumDataPublisher.
    1. Open ConnectorConfig.xml
    2. Go to <AlarmConfiguration> section
    3. Add alarm filter name to <AlarmFilterName>/<AlarmFilterName>
  7. Start the SpectrumDataPublisher.
Restart the SpectrumDataPublisher, when you change the alarm filters in the OneClick client.
Troubleshooting SpectrumDataPublisher
Troubleshooting information for the problems that are encountered with SpectrumDataPublisher.
DX NetOps Spectrum
Data does not synchronize to DX Operational Intelligence
Symptom:
SpectrumDataPublisher.log shows the exception: com.ca.spectrum.spub.common.ConnectorException.
DX NetOps Spectrum
Data does not synchronize to DX Operational Intelligence.
Resolution:
Provide correct details of the Jarvis server under the DestinationConfiguration section of ConnectorConfig.xml and make sure the DestinationHostname is resolved to a valid IP address.