Upgrade Fault Tolerant Data Aggregators

To upgrade the Data Aggregators in an existing fault tolerant environment, complete the following steps:
capm350
To upgrade the Data Aggregators in an existing fault tolerant environment, complete the following steps:
2
If you enabled FIPS-compliant encryption, and you upgrade
Performance Center
before the Data Aggregator, temporary FIPS compatibility synchronization errors occur. These errors occur until the Data Aggregator is upgraded. To avoid this temporary condition, upgrade the Data Aggregator before upgrading
Performance Center
.
If you are setting up fault tolerance for the first time, see Fault Tolerance.
Verify the Prerequisites
Meet the following prerequisites before you upgrade the Data Aggregator:
  • Verify that ports 8581, 61616, 61618, 61620, and 61622 are open for communication between your Data Collectors and your Data Aggregators.
    For the detailed port list, Review Installation Requirements and Considerations.
  • Before you upgrade, verify the zip and unzip packages are installed. If these packages are not installed, use the following command to install them:
    yum -y install zip unzip
  • For the Data Aggregator and the Data Collectors, the default maximum memory is 80% of the total system memory. AMQ uses 20% of total memory on both components. Both components reserve 2 GB of memory for the operating system. These maximum values can be modified, however the modifications are not preserved during an upgrade. For more information, see Modify Maximum Memory Usage for Data Aggregator and Data Collector Components.
  • Ensure that you have a shared data directory (example: 
    /DASharedRepo
    ) and that the same user ID is shared between Data Aggregator hosts. Data from whichever Data Aggregator is active is stored in this directory.
    For information about the sizing requirements, see the capm Sizing Tool.
    If you are using NFS, only NFS 4 and higher is supported because of the ActiveMQ Kaha locking requirements.
    The shared data directory must be accessible at all times. If the shared data directory is down and is inaccessible, no data is loaded and data loss occurs.
Verify the Limit on the Number of Open Files on Data Aggregator
Verify that the user that is installing Data Aggregator has a limit of at least 65536 on the number of open files. Set this value permanently.
Follow these steps:
  1. As the root user or a sudo user, log in to the Data Aggregator host.  
  2. Change the ulimit for the open files limit to at least 65536:
    ulimit -n 
    ulimit_number
    For example:
    ulimit -n 65536
  3. Open the following file:
    /etc/security/limits.conf
  4. Add the following lines:
    # Added by Data Aggregator
    * soft nofile 65536
    # Added by Data Aggregator
    * hard nofile 65536
    Restart Data Aggregator for these changes to take effect. If you are upgrading, the upgrade process automatically restarts Data Aggregator.
  5. Verify that the number of open files is set properly:
    ulimit -n
    The command returns the limit that you specified earlier.
Verify That the Data Repository Is Running
The Data Aggregator Upgrade requires that the Data Repository is running.
Follow these steps:
  1. Log in to the Data Repository host.
  2. Verify that the Data Repository is running:
    /opt/vertica/bin/vsql -U 
    dauser
     -w 
    dapass
     -c 'select version()'
    The expected return shows the version of the Data Repository:
                  version
    ------------------------------------
     Vertica Analytic Database vx.x.x-x
(Optional) Temporarily Connect to Older Versions of Data Collectors
You can reduce data loss during an upgrade by temporarily connecting existing Data Collectors (version 3.0, 3.1, or 3.2) to an upgraded Data Aggregator. Consider this upgrade option if you have many Data Collectors to upgrade.
Important! Connecting 3.x versions of Data Collectors to upgraded Data Aggregators is only supported on a temporary basis.
New features in
CA Performance Management
3.5 may not be compatible with older Data Collectors. Before using new features or making configuration changes, upgrade all Data Collectors to the same version as the Data Aggregator.
Follow these steps before upgrading the Data Aggregator:
 
  1. Create a file called com.ca.im.dm.core.collector.controller.cfg in the following location: 
    /opt/IMDataAggregator/apache-karaf-2.4.3/etc/
  2. Add the following single line to the file: 
    allowed-mismatch-dc-version=3.0.0.0
  3. Save the file. 
  4. Proceed with upgrading the Data Aggregator.
Upgrade the Proxy Server
Before you upgrade your existing fault tolerant Data Aggregators, upgrade the proxy server.
Follow these steps:
  1. Uninstall the proxy server. For more information, see Install or Uninstall the Proxy Server.
  2. (RHEL 7 only) Run the following command on the proxy server host as the root or sudo user:
    systemctl daemon-reload
  3. Run the following command on the proxy server host as the root or sudo user:
    service consul stop
  4. Delete the proxy installation directory:
    rm -rf daproxy/
  5. Install the proxy server. For more information, see Install or Uninstall the Proxy Server.
Upgrade Fault Tolerant Data Aggregators
If you are upgrading Data Aggregators in an existing fault tolerant environment, do the following upgrade steps in the following order:
  1. Put the inactive Data Aggregator(B) into maintenance mode.
  2. Upgrade the inactive Data Aggregator(B).
  3. Activate the inactive Data Aggregator(B).
    This Data Aggregator(B) becomes available for failover.
  4. Put the active Data Aggregator(A) into maintenance mode.
    This step causes failover. The previously inactive Data Aggregator(B) becomes the active Data Aggregator.
  5. Upgrade the now inactive Data Aggregator(A).
  6. Activate the now inactive Data Aggregator(A).
    This Data Aggregator(A) becomes available for failover. The other Data Aggregator(B) is the active Data Aggregator.
Turn on Maintenance Mode
If you are upgrading the Data Aggregators in an existing 3.5 GA fault tolerant environment, you must put the Data Aggregator into maintenance mode before upgrade.
For more information, see Fault Tolerance.
Follow these steps:
  1. Log in to the Data Aggregator host as the root user or a sudo user.
  2. Run one of the following commands to shut down the inactive Data Aggregator and prevent it from restarting until the upgrade is complete:
    • RHEL 6.x:
      service dadaemon maintenance
    • RHEL 7.x:
      DA_Install_Directory
      /scripts/dadaemon maintenance
Upgrade the Data Aggregator
If you are upgrading the Data Aggregators in an existing 3.5 GA fault tolerant environment, you must put the Data Aggregator into maintenance mode before upgrade.
Upgrade the inactive Data Aggregator first and then activate it. Then upgrade and activate the active Data Aggregator.
Follow these steps:
  1. Log in to the Data Aggregator host as the root user or a sudo user.
  2. Copy the installDA.bin file to the /tmp directory.
  3. Change to the /tmp directory:
    cd /tmp
  4. Change permissions for the installation file:
    chmod a+x installDA.bin
  5. To run the upgrade, do one of the following steps:
    • Run the installation as the root user:
      ./installDA.bin -i console
    • Run the installation as the sudo user:
      sudo ./installDA.bin -i console
      For environments that support a GUI, remove the following argument to run the upgrade in GUI mode:
      -i console
      To generate a response file for silent upgrade, add the following argument:
      -r 
      response_file
      response_file_directory
      specifies the directory the directory path and file name for the response file.
      Example:
      /temp/installer.properties
      To run the upgrade in silent mode, use the following command:
      ./installDA.bin -i silent -f 
      response_file
      response_file
      is the directory path and file name of the previously generated response file.
  6. Follow the instructions in the console.
    If the Data Repository was migrated, follow the Data Aggregator upgrade instructions in the console. Answer NO when prompted to drop the schema and the upgrade continues. If you answer YES, you lose your migrated data.
    Data Aggregator is upgraded. The console displays a confirmation message.
  7. Verify access to the Data Aggregator REST endpoints. Open a Web browser on a computer with HTTP access to Data Aggregator. Navigate to the following address:
    http://
    DA_Proxy
    :
    port
    /rest
    DA_Proxy:port
    Specify the Data Aggregator proxy host name and the required port number.
    The return is a list of hyperlinks for available web services. When you click a link, the XML content describing the selection displays.
    Table segmentation is required for systems where the original installation was a release earlier than 2.3.3. Segmentation is a one-time task that improves database performance and reduces the required disk space. If you received a message during the upgrade that the database tables require segmentation, complete table segmentation after the upgrade. For more information, see Segment Database Tables.
    The Karaf log in the Data Aggregator includes the following error after upgrading the installation:
    ERROR | tenderThread-178 | 2013-01-24 13:36:40,431 | ndorCertificationPriorityManager | nager.core.cert-mgr.impl |
          | Failed to load the MetricFamilyVendorPriority for bundle: BundleURLEntry [bundle=198, resourceURL=file:/opt/IMDataAggregator/apache-karaf-<
    vers
    >/data/cache/resources/198--xml-vendorpriorities-ReachabilityVendorPriorities.xml
    This error for Reachability is expected and harmless. Other occurrences of this error are not expected.
  8. (Optional) In the
    Performance Center
    user interface, hover over 
    Administration
    , select 
    Data Sources: System Status
    , and verify the status of the Data Aggregator.
Activate the Data Aggregator
If you upgraded the Data Aggregators in an existing fault tolerant environment, you must activate each Data Aggregator after upgrade.
For more information, see Fault Tolerance.
The
Data Aggregator may take several minutes to start.
Follow these steps:
  1. Log in to the the Data Aggregator host that is in maintenance mode as the root user or sudo user.
  2. Run one the following commands to enable the fault tolerant Data Aggregator so that it can start when necessary:
    • RHEL 6.x:
      service dadaemon activate
    • RHEL 7.x:
      DA_Install_Directory
      /scripts/dadaemon activate