Upgrade Fault-Tolerant Data Aggregators

You can upgrade data aggregators in existing fault-tolerant environments.
Use the following procedures to upgrade data aggregators in existing fault-tolerant environments.
In this article:
2
3
Verify the Prerequisites
Ensure that you have met the following prerequisites before you upgrade the data aggregator:
  • If you are setting up fault tolerance for the first time, you have reviewed Fault Tolerance.
  • You have verified that ports TCP 8581, 61616, 61618, 61620, and 61622 are open to allow communication between the data collectors and the data aggregators.
    For more information about the ports that are required for
    DX NetOps Performance Management
    communications, see Review Installation Requirements and Considerations.
  • You have verified that you have installed the zip and unzip packages. If these packages are not installed, use the following command to install them:
    yum -y install zip unzip
  • For the data aggregator and the data collectors, you have verified that the maximum memory usage for data aggregator and data collector. The default maximum memory is 80% of the total system memory. AMQ uses 20% of total memory on both components. Both components reserve 2 GB of memory for the operating system. You can modify these maximum values, however the modifications are not preserved during an upgrade.
    For more information, see Modify Maximum Memory Usage for Data Aggregator and Data Collector Components.
  • You have ensured that you have a shared data directory (example:
    /DASharedRepo
    ) and that the same user ID is shared between data aggregator hosts. Data from whichever data aggregator is active is stored in this directory.
    For information about the sizing requirements, see the
    DX NetOps Performance Management
    Sizing Tool
    .
    If you are using NFS, only NFS 4 and higher is supported because of the ActiveMQ Kaha locking requirements.
    The shared data directory must be accessible at all times. If the shared data directory is down and is inaccessible, no data is loaded and data loss occurs.
  • If you are upgrading the fault-tolerant data aggregators in an existing
    DX NetOps Performance Management
    3.5 fault-tolerant environment, you have put the data aggregator into maintenance mode.
Verify the Limit on the Number of Open Files on the Data Aggregator
Verify that the user that is installing the data aggregator has a limit of at least 65536 on the number of open files. Set this value permanently.
Follow these steps:
  1. As the root user or a sudo user, log in to the data aggregator host.
  2. Change the ulimit for the open files limit to at least 65536 by issuing the following command:
    ulimit -n
    ulimit_number
    For example:
    ulimit -n 65536
  3. Open the
    /etc/security/limits.conf
    file.
  4. Add the following lines:
    # Added by Data Aggregator
    * soft nofile 65536
    # Added by Data Aggregator
    * hard nofile 65536
    These changes to take effect when the data aggregator restarts. The upgrade process automatically restarts the data aggregator.
  5. Verify that the number of open files is set properly by issuing the following command:
    ulimit -n
    The command returns the limit that you specified earlier.
Verify that the Data Repository is Running
The data aggregator upgrade requires that the data repository is running.
Follow these steps:
  1. Log in to the data repository host.
  2. Verify that the data repository is running by issuing the following command:
    /opt/vertica/bin/vsql -U
    dauser
    -w
    dapass
    -c 'select version()'
    The following response is expected:
    version
    ------------------------------------
    Vertica Analytic Database vx.x.x-x
Upgrade the Proxy Server
Follow these steps:
  1. Uninstall the proxy server (DAProxy).
    For more information, see Install or Uninstall the Proxy Server.
  2. (RHEL 7.x, SLES, and OL only) Issue the following command on the proxy server host as the root or sudo user:
    systemctl daemon-reload
  3. Stop the Consul service on the proxy server by issuing the following command as the root or sudo user:
    service consul stop
  4. Delete the proxy installation directory by issuing the following command:
    rm -rf daproxy/
  5. Install the proxy server.
    For more information, see Install or Uninstall the Proxy Server.
Upgrade the Fault-Tolerant Data Aggregators
Use the following process to upgrade a set of fault-tolerant data aggregators:
  1. Put Data Aggregator(B) into maintenance mode.
    For more information, see Turn on Maintenance Mode.
    Data Aggregator(B) becomes inactive.
  2. Upgrade the inactive Data Aggregator(B).
    For more information, see Upgrade the Data Aggregator.
  3. Activate the inactive Data Aggregator(B).
    For more information, see Activate the Data Aggregator.
    Data Aggregator(B) is now active and available for failover.
  4. Put the active Data Aggregator(A) into maintenance mode.
    For more information, see Turn on Maintenance Mode.
    Data Aggregator(A) becomes inactive.
    This step causes failover. The previously inactive Data Aggregator(B) becomes the active data aggregator.
  5. Upgrade the now-inactive Data Aggregator(A).
    For more information, see Upgrade the Data Aggregator.
  6. Activate inactive Data Aggregator(A).
    For more information, see Activate the Data Aggregator.
    Data Aggregator(A) is now active and becomes available for failover. Data Aggregator(B) is the active data aggregator.
Turn on Maintenance Mode
Follow these steps:
  1. Log in to the data aggregator host as the root user or a sudo user.
  2. Shut down the inactive data aggregator and prevent it from restarting until the upgrade is complete by issuing one of the following commands:
    • RHEL 6.x:
      service dadaemon maintenance
    • RHEL 7.x, SLES, or OL:
      DA_Install_Directory
      /scripts/dadaemon maintenance
The data aggregator becomes inactive.
Upgrade the Data Aggregator
Upgrade the inactive data aggregator first, and then activate it. Then upgrade and activate the active data aggregator.
Follow these steps:
  1. Log in to the data aggregator host as the root user or a sudo user.
  2. Copy the
    installDA.bin
    file to the
    /tmp
    directory.
  3. Change to the
    /tmp
    directory by issuing the following command:
    cd /tmp
  4. Change permissions for the installation file by issuing the following command:
    chmod a+x installDA.bin
  5. To run the upgrade, do one of the following steps:
    • Run the installation as the root user by issuing the following command:
      ./installDA.bin -i console
    • Run the installation as the sudo user by issuing the following command:
      sudo ./installDA.bin -i console
      To generate a response file for silent upgrade, add the following argument:
      -r
      response_file
      response_file_directory
      specifies the directory the directory path and file name for the response file.
      Example:
      /temp/installer.properties
      To run the upgrade in silent mode, use the following command:
      ./installDA.bin -i silent -f
      response_file
      response_file
      is the directory path and file name of the previously generated response file.
  6. Follow the data aggregator upgrade instructions in the console.
    If you have migrated the data repository, to protect your migrated data, to drop the schema, and to continue the upgrade, answer
    NO
    when prompted.
    The data aggregator is upgraded. The console displays a confirmation message.
  7. Verify access to the data aggregator REST endpoints. Open a Web browser on a computer with HTTP access to the data aggregator. Navigate to the following address:
    http://
    DA_Proxy
    :
    port
    /rest
    DA_Proxy:port
    Specify the data aggregator proxy host name and the required port number.
    A list of hyperlinks for available web services is returned. Click a link to display the XML content describing the selection.
    If you receive a message that the database tables require segmentation, complete this step after the upgrade. Table segmentation is required for systems where the original
    DX NetOps Performance Management
    installation was
    DX NetOps Performance Management
    2.3.2 or earlier. Table segmentation is a one-time task that improves database performance and reduces the required disk space.
    For more information about how to segment database tables, see Segment Database Tables.
    The Karaf log in the data aggregator includes the following error after upgrading the installation:
    ERROR | tenderThread-178 | 2013-01-24 13:36:40,431 | ndorCertificationPriorityManager | nager.core.cert-mgr.impl |
    | Failed to load the MetricFamilyVendorPriority for bundle: BundleURLEntry [bundle=198, resourceURL=file:/opt/IMDataAggregator/apache-karaf-<
    vers
    >/data/cache/resources/198--xml-vendorpriorities-ReachabilityVendorPriorities.xml
    This Reachability error is expected and harmless. Other occurrences of this error are not expected.
  8. (Optional) Verify the status of the data aggregator:
    1. In
      NetOps Portal
      , hover over
      Administration
      ,
      Data Sources
      , and then select
      System Status
      .
      The
      System Status
      page appears.
    2. Verify the status of the data aggregator.
      The
      Polling Status
      column on this page might inaccurately display "Not Connected" for the data collectors.
      For more information about this known issue, see Known Limitations.
Activate the Data Aggregator
If you upgraded the data aggregators in an existing fault-tolerant environment, activate each data aggregator.
For more information, see Fault Tolerance.
The data aggregator might take several minutes to start.
Follow these steps:
  1. Log in to the data aggregator host that is in maintenance mode as the root user or sudo user.
  2. Enable the fault tolerant data aggregator so that it can start when necessary by issuing one of the following commands:
    • RHEL 6.x:
      service dadaemon activate
    • RHEL 7.x, SLES, or OL:
      DA_Install_Directory
      /scripts/dadaemon activate
The data aggregator becomes active and available for failover.
Point the Data Aggregator Data Source to the Proxy Server
Complete this step only if you are converting a single (non-fault-tolerant) data aggregator to a fault-tolerant environment. Point the data aggregator data source in
NetOps Portal
to the proxy server.
Follow these steps:
  1. Edit the data aggregator data source in
    NetOps Portal
    .
    For more information about how to edit a data aggregator data source, see Configure a Data Source.
  2. On the
    Edit Data Source
    dialog, in the
    Host Name
    field, enter the host name for the proxy server, and then save your changes.