Rehydrating Data in a Cloud Environment

If you have
DX NetOps Performance Management
set up in an cloud environment, you can patch operating systems from a common image instead of patching each operating system individually. The following procedures outline the necessary steps for rehydrating the
DX NetOps Performance Management
nodes with minimal data loss.
capm370
If you have
DX NetOps Performance Management
set up in a cloud environment, you can patch operating systems from a common image instead of patching each operating system individually. The following procedures outline the necessary steps for rehydrating the
DX NetOps Performance Management
nodes with minimal data loss.
Verify the Prerequisites
Before rehydration, ensure your environment is in a good state.
Follow these steps:
  1. Select
    Administration
    ,
    Data Source Settings
    , and
    System Status
    .
  2. Verify that the Data Aggregator and Data Repository are connected.
  3. Verify that the Data Aggregator is up and running.
  4. View the System Status page to verify there is no backed up or cached poll data.
    If the PRQ queue is not empty, the Data Collectors need to send the rest of the polled data to the Data Aggregator. The queue fills up when an outage occurs, which causes data to be cached on the Data Collectors. View the System Status page to verify that all the Data Collectors have a green status. The Polling Status column shows whether cached values exist on the Data Collector.
  5. If you have
    DX NetOps Virtual Network Assurance
    in your environment, do the following steps:
    1. In
      NetOps Portal
      , hover over
      Administration
      , and click
      Monitored Items Management: VNA Gateways
      .
    2. Set the
      Administrative Status
      to
      Down
      .
Rehydrate Each Data Collector
Rehydrate each Data Collector one at a time. Make sure each Data Collector recovers and starts polling before you rehydrate each Vertica node and the Data Aggregator. During this process, some polls are cached on the Data Collectors for a time.
Follow these steps:
  1. Build the new operating system on the Data Collector container or virtual machine.
  2. Copy the
    DCM_ID
    by issuing the following command:
    grep "manager\-id\="
    DC_Install_Directory
    /apache-karaf-<
    version>
    /etc/com.ca.im.dm.core.collector.cfg
  3. Bring down the old container or virtual machine.
  4. Give the new container or virtual machine a new IP address or name and bring it online.
  5. Reinstall the Data Collector with the
    DCM_ID
    of the original Data Collector by issuing the following commands:
    export DCM_ID="
    Original_DC_Host
    :
    DCM_ID
    "
    cd /tmp;
    rm -rf install.bin;
    wget http://
    DA_Host
    :
    Port
    /dcm/InstData/Linux/VM/install.bin;
    chmod a+x install.bin;
    ./install -i silent
    The Data Collector installs, reconnects to the Data Aggregator, and starts polling.
Rehydrate Each Vertica Node
Rehydrate each Vertica node one at a time. Before you start, verify that all nodes are up and running:
/opt/vertica/bin/admintools -t list_allnodes
Follow these steps:
  1. Bring down the Vertica node by issuing the following command:
    /opt/vertica/bin/admintools -t stop_node -s
    IP_Address
  2. Unmount the
    data
    directory and the
    catalog
    directory.
  3. Create a new node with the same IP address and name.
  4. Mount the
    data
    directory and the
    catalog
    directory to the new node.
  5. Run the validation script:
    ./dr_validate.sh -n -p drinstall.properties
    The script validates the system settings. Review and resolve any errors or warnings. You can run this script multiple times to verify that all system configuration options are set properly. The validation script may prompt you to reboot.
  6. Install Vertica from an up and running node by issuing the following command:
    /opt/vertica/sbin/install_vertica -u
    dradmin
    -l /export/dradmin -d /export/data -L ./resources/vlicense.dat -Y -r ./resources/vertica-<
    version>
    .rpm
    Values should match those in the properties files for
    dr_install.sh
    , and point to the same resources.
  7. Start the node and verify that it is up and running by issuing the following command:
    /opt/vertica/bin/admintools -t restart_node -s
    Host_Name
    -d
    DB_Name
    The state starts as DOWN, then changes to REBUILDING until it changes to UP.
  8. Repeat this procedure for each node.
  9. After all nodes are rehydrated, verify that all nodes are back up and running as the
    dradmin
    user by issuing the following command:
    /opt/vertica/bin/admintools -t list_allnode
Rehydrate the Data Aggregator
During this Vertica refresh, the Data Aggregator should collect data from the Data Collectors. The Data Aggregator pushes data to Vertica the entire time on the up and running nodes. The speed of the ingestion is sometimes cut in half during this time. After you rehydrate the Data Collectors and Vertica, you can rehydrate the Data Aggregator.
Follow these steps:
  1. Prepare a new node for the Data Aggregator.
  2. Do one of the following steps:
    • Stop the Data Aggregator service by issuing the following command:
      service dadaemon stop
      For RHEL 7.x,
      service
      invokes
      systemctl
      . You can use
      systemctl
      instead.
    • (Fault tolerant environment) If the local Data Aggregator is running, issue one the following commands to shut it down and prevent it from restarting until maintenance is complete:
      • RHEL 6.x:
        service dadaemon maintenance
      • RHEL 7.x, SLES, or OL:
        DA_Install_Directory
        /scripts/dadaemon maintenance
    The Data Aggregator completes processing and the service stops.
  3. Move the configuration files to the new node.
    For more information, see Back Up Data Aggregator.
    In environments with fault tolerant Data Aggregators, use the shared data directory and reattach it. For more information, see Fault Tolerance.
  4. Do one of the following steps:
    • Start the Data Aggregator service by issuing the following command:
      service dadaemon start
    • (Fault tolerant environment) Run one the following commands to enable the fault tolerant Data Aggregator so that it can start when necessary:
      • RHEL 6.x:
        service dadaemon activate
      • RHEL 7.x, SLES, or OL:
        :
        DA_Install_Directory
        /scripts/dadaemon activate
    The Data Aggregator consumes the queued polls and pushes them to Vertica.
    Depending on the total outage time, all cached data is consumed and ready for reporting in approximately two times the outage time. When the ActiveMQ consumption returns to normal, this indicates there is no longer a backlog. View the System Status page to verify that all the Data Collectors have a green status and that the system is receiving approximately the same number of polls as it was before the process started.
Rehydrate
NetOps Portal
Finally, you can rehydrate
NetOps Portal
.
Follow these steps:
  1. Prepare a node for
    NetOps Portal
    .
  2. Bring down
    NetOps Portal
    by issuing the following commands:
    For RHEL 7.x and higher or OL,
    service
    invokes
    systemctl
    . You can use systemctl instead.
    service caperfcenter_console stop service caperfcenter_devicemanager stop service caperfcenter_eventmanager stop service caperfcenter_sso stop
  3. Move the database to the new node.
    For more information, see Back Up Performance Center.
  4. Start
    NetOps Portal
    :
    1. Start the SSO service by issuing the following command:
      service caperfcenter_sso start
    2. Wait one minute, then start the event manager and device manager by issuing the following commands:
      service caperfcenter_eventmanager start service caperfcenter_devicemanager start
    3. Wait one minute, then start the console service:
      service caperfcenter_console start
Rehydrate
DX NetOps Virtual Network Assurance
If you have
DX NetOps Virtual Network Assurance
in your environment, rehydrate it now.
Follow these steps:
  1. Query the following REST URL to find the engine ID required later:
    http://
    VNA_host
    :8080/vna/rest/v1/admin/engines
  2. Query the following REST URL for your plug-in configuration required later:
    http://
    VNA_host
    :8080/vna/rest/v1/admin/engines/
    Engine_ID
    /config
  3. Stop WildFly using one of the following commands:
    service wildfly stop
    systemctl stop wildfly
  4. Back up the existing database:
    VNA_Install_Directory
    /tools/bin/db_backup.sh
    Backup_File_Name
  5. Install
    DX NetOps Virtual Network Assurance
    on the new server and restore the database from the backup:
    VNA_Install_Directory
    /tools/bin/db_restore.sh
    Backup_File_Name
  6. Reconfigure the plug-ins using the information from your original query.
    Use the same Domain ID from the original configuration.
  7. In
    NetOps Portal
    , do the following steps:
    1. Hover over
      Administration
      , and then click
      Monitored Items Management: VNA Gateways
      .
    2. Change the new
      DX NetOps Virtual Network Assurance
      server ID.
    3. Set the
      Administrative Status
      to
      Up
      .
Reconnect an Existing
DX NetOps Spectrum
Data Source
If you rehydrate the
DX NetOps Performance Management
environment, and want to reconnect it to an existing Spectrum server, complete the following procedure.
Follow these steps:
  1. Go to
    Administration
    ,
    Data Sources
    ,
    Data Sources
    .
  2. Select the
    DX NetOps Spectrum
    data source, and click
    Edit
    .
  3. Change the Status to Disabled, and click
    Save
    .
  4. Remove the
    NetOps Portal
    entries:
    cd
    Spectrum_Install_Directory
    /vnmsh ./connect ./show models | grep CAPC
  5. Run the following command for every
    CAPCIPDomain
    and
    CAPCTenant
    model found.
    ./destroy model mh=0xXXXXX
  6. Remove the
    NetOps Portal
    integration database from
    DX NetOps Spectrum
    :
    bash -login
    cd mysql
    cd bin
    ./mysqladmin --defaults-file=../my-spectrum.cnf -unetqos -p
    password
    drop netqos_integ
  7. Restart
    DX NetOps Spectrum
    tomcat:
    cd
    Spectrum_Install_Directory
    /tomcat/bin
    ./stopTomcat.sh
    ./startTomcat.sh
  8. Go to
    Administration
    ,
    Data Sources
    ,
    Data Sources
    .
  9. Select the
    DX NetOps Spectrum
    data source, and click
    Edit
    .
  10. Change the Status to Enabled, and click
    Save
    .
Rehydrate
DX NetOps Network Flow Analysis
If you have
DX NetOps Network Flow Analysis
in your environment, rehydrate it now.
Follow these steps:
  1. Go to
    Administration
    ,
    Data Sources
    ,
    Data Sources
    .
  2. Select the
    DX NetOps Network Flow Analysis
    data source, and click
    Edit
    .
  3. Change the Status to Disabled, and click
    Save
    .
  4. Determine the database files to backup.
    • Customized data_retention database (Stand-alone or Harvester server):
      data_retention
    • harvester database (Stand-alone or Harvester server):
      harvester
    • reporter database (Stand-alone or NFA console):
      reporter
  5. Copy each of the target directories or files to a remote location.
  6. Back up the following databases to a remote location, using
    mysqldump
    . Back up the
    reporter
    database last, regardless of the deployment architecture.
    • Customized data_retention database (Stand-alone or Harvester server):
      data_retention
    • harvester database (Stand-alone or Harvester server):
      harvester
    • reporter database (Stand-alone or NFA console):
      reporter
    mysqldump --routines --events -u root
    dbname
    --skip-lock-tables >
    dbbackupname
    .sql
  7. (Optional) Verify that the
    mysqldump
    was successful by checking that the size of the backup is over 1 KB.
  8. Restore each of the target directories or files from its remote location to its original location.
  9. Restore each of the databases. Restore the reporter database first, regardless of the deployment architecture.
    • reporter database:
      reporter
      (Stand-alone or NFA console)
    • Customized data_retention database:
      data_retention
      (Stand-alone or Harvester server)
    • harvester database:
      harvester
      (Stand-alone or Harvester server)
      For best results, restore to a clean installation.
      mysql –e “drop database
      DB_Name
      ;”
      mysql -e "create database
      DB_Name
      ;"
      mysql -u root
      DB_Name
      dbbackupname
      .sql
      mysql -u root mysql > proc.sql
  10. Go to
    Administration
    ,
    Data Sources
    ,
    Data Sources
    .
  11. Select the
    DX NetOps Network Flow Analysis
    data source, and then click
    Edit
    .
  12. Change the
    Status
    to
    Enabled
    , and then click
    Save
    .