Migrate the Data Repository

Migrate the data repository to another cluster if:
  • You are moving to new hosts for a major operating system (OS) upgrade (for example, Red Hat Enterprise Linux (RHEL) 6.9 to RHEL 8.1).
  • The current database hardware no longer meets sizing requirements.
  • You are moving from virtual machines to physical hardware for the database.
Use the following process to migrate the data repository:
2
Depending on the amount of data, migrating the data repository can take a significant amount of time.
To minimize downtime, perform incremental backups after your initial backup.
Verify the Prerequisites
Before migrating the data repository, ensure that you have completed the following prerequisite steps:
  • You understand the process of copying databases from one cluster to another.
    For more information about copying the database to another cluster, see the Vertica documentation.
  • You have upgraded the existing system to the
    NetOps Portal
    version to which you are migrating.
For more information about how to upgrade
NetOps Portal
, see Upgrade
NetOps Portal
.
Prepare to Install the Destination (Target) Data Repository
Before you install Vertica on a new data repository cluster (the target cluster), prepare the environment for the installation.
For more information, see Prepare to Install the Data Repository.
The target cluster must have the following things:
  • The same number of nodes (node count), name (database name), and node names the source cluster.
    • The target cluster can be empty.
    • The nodes names that are listed in the
      NODES
      system tables on both clusters must match.
      For more information about how to change node names post install on the new system to match the existing one, contact Support.
  • Be accessible from the source cluster.
  • The same database administrator account and all nodes must allow a database administrator of the source cluster to log in through SSH without a password.
    Passwordless access within a cluster is not the same as passwordless access between clusters. The SSH ID of the administrator account on the source cluster and the target cluster are likely not the same. Configure each host in the target cluster to accept the SSH authentication of the source cluster.
  • Adequate disk space for the
    vbr --task copycluster
    command to complete.
Install Vertica on the New Cluster
The installation process is the same as a normal data repository installation.
For more information, see Install the Data Repository.
When installing, use the same configuration for the target cluster as for the source cluster. For example, the Vertica version, node count, database name, administrator, user, catalog directory, and data directory must be the same as the original source data repository.
For more information, see the "Prepare to Install the Destination (Target) Data Repository" section.
Migrate the Cluster
After you install the database (Vertica) on the new (target) cluster, migrate the existing source data. The migration uses the
copycluster
command, which simultaneously backs up the existing source database and restores the data to the new target cluster. The command configures, starts, and runs the copy cluster on the source system to point to the new target system. You can issue this command on a single node on the source system and it copies over to all nodes on the new system. The command copies the data in the data repository from before you run the command. Because
DX NetOps Performance Management
continues to collect data during the migration, the process requires that you issue the command multiple times.
Use the following process to migrate the cluster:
3
Create a Configuration File for Copy Cluster
Before you can configure the target cluster, you need to know the exact names that
admintools
supplied to all nodes in the source database.
To see the node names, run a query similar to following example:
VMart=> select node_name from nodes;
node_name
------------------
v_vmart_node0001
v_vmart_node0002
v_vmart_node0003
(3 rows)
The
copycluster
command requires a configuration file that includes the necessary information. You can create the file in any desired location on the source system and give it any name with a
.ini
extension. In the configuration file, specify the host names of nodes in the target cluster as the backup hosts. Specify the nodes in a
[Mapping]
section as shown in the following example. Specify the full node name, not the short name (for example,
v_vmart_node0001
, not
node0001
).
Example:
The following example configuration file is set up to copy a database on a three node cluster (
v_vmart_node0001
,
v_vmart_node0002
, and
v_vmart_node0003
) to another cluster consisting of nodes (
test-host01
,
test-host02
, and
test-host03
):
The
dbName
parameter is case-sensitive.
[Misc]
snapshotName = CopyVmart
restorePointLimit = 5
objectRestoreMode = createOrReplace
tempDir = /tmp/vbr
retryCount = 5
retryDelay = 1
[Database]
dbName =
vmart
dbUser =
dradmin
dbPassword =
password
dbPromptForPassword = False
[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
[Mapping]
; backupDir is not used for cluster copy
v_vmart_node0001= test-host01
v_vmart_node0002= test-host02
v_vmart_node0003= test-host03
Ensure that the
/tmp/vbr
directory has read and write permissions.
Stop the Target Database
Before you start the migration, shut down the database on the target database cluster.
Follow these steps:
  1. Log in to the target database cluster as the database admin user.
  2. Start the Vertica Administration Tools utility:
    /opt/vertica/bin/adminTools
  3. Select
    (4) Stop Database
    .
    Wait for the shutdown to complete.
Copy the Historical Data
After you install the database on the new (target) cluster, copy the data from the existing (source) database. The
copycluster
command copies the data from
before
you initiate the command. New data continues to come in while the command is running, but this data is not copied.
The
copycluster
command requires that passwordless SSH is enabled for the database administrator account. You enabled this when you set up the data repository.
Follow these steps:
  1. Log in to the source cluster with the database administrator account.
  2. Ensure that passwordless SSH is enabled (the passwordless ssh key is set up) for the database administrator account by issuing the following command. In a cluster installation, ensure that passwordless ssh keys are set up for
    each
    host that is participating in the cluster:
    ssh
    hostname
    ls
    • hostname
      The name of the host where the data repository is installed.
    If the passwordless ssh key is set up for the database administrator account, you are
    not
    prompted for a password. If you are prompted for a password, set up the passwordless ssh key for the database administrator account by completing the following steps:
    1. Press the
      Ctrl + C
      keys on your keyboard.
    2. Set up the Linux user account for the database administrator user with a passwordless ssh key by issuing the following command:
      ssh-keygen -N "" -t rsa -f ~/.ssh/id_rsa
      cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys2
      chmod 644 ~/.ssh/authorized_keys2
    3. Copy the SSH credentials to the other hosts in the cluster by issuing the following command:
      ssh-copy-id -i
      dradmin
      @
      other-hostname-in-the-cluster
    4. Confirm that you are
      not
      prompted for a password by issuing the following command:
      ssh hostname ls
  3. Copy the historical data for the source database by issuing the following command:
    vbr.py --task copycluster --config-file
    CopyClusterConfigurationFile
    .ini
    The following message is displayed:
    vbr.py --config-file CopyVmart.ini --task copycluster
    Preparing...
    Copying...
    1871652633 out of 1871652633, 100%
    All child processes terminated successfully.
    copycluster done!
The historical data is copied.
Verify the Copy of the Historical Data
After the copy cluster process completes, ensure the integrity of your data.
Follow these steps:
  1. Log in to the target database cluster as the database admin user.
  2. Start the Vertica Administration Tools utility:
    /opt/vertica/bin/adminTools
  3. Select
    (3) Start Database
    .
  4. From any node in the cluster, open the Vertica SQL prompt by issuing the following command:
    /opt/vertica/bin/vsql -U dauser
  5. Verify the timestamp of these key database tables by issuing the following queries:
    SELECT to_timestamp(max(tstamp)) from dauser.reach_rate;
    SELECT to_timestamp(max(tstamp)) from dauser.ifstats_rate;
    The date and time must correspond to the time when you started the copy.
Start the Target Database
Start the database on the target database cluster.
Follow these steps:
  1. While logged in to the target database cluster as the database admin user, start the Vertica Administration Tools utility:
    /opt/vertica/bin/adminTools
  2. Select
    (3) Start Database
    .
Stop the Data Aggregator
To maintain integrity of your data, stop the data aggregator before the final copy of the data repository. Polling continues on the data collector if it is running and polling when the data aggregator is stopped. During this phase, the data collector continues to receive and queue poll responses from the managed devices. After copy cluster is done and you have restarted the data aggregator and unblocked the data collector firewall rules, the data collector delivers these polled responses to the data repository.
Follow these steps:
  1. Log in to the data aggregator host as the root user or a sudo user.
    If you installed the data aggregator as the sudo user, you set up a sudo command alias for the service
    dadaemon
    command. Use the sudo commands.
  2. Use firewall rules to block traffic from the data collectors. Issue the following command for each data collector:
    iptables -A INPUT -s
    DC_IP
    -j DROP
    • DC_IP
      The IP of the data collector.
  3. Do one of the following steps:
    • Issue the following command:
      service dadaemon stop
      For RHEL 7.x or OL, the
      service
      command invokes the
      systemctl
      command. You can use
      systemctl
      instead.
    • (Fault-tolerant environments) If the local data aggregator is running, shut it down and prevent it from restarting until maintenance is complete by issuing one the following commands:
      • RHEL 6.x:
        service dadaemon maintenance
      • RHEL 7.x/8.x, SUSE Linux Enterprise Server (SLES), or Oracle Linux (OL):
        <installation_directory>
        /IMDataAggregator/scripts/dadaemon maintenance
        • installation_directory
          The installation directory for the data aggregator.
          Default:
          /opt
The data aggregator stops.
Copy Recent Data
Copy recent data in the source cluster by issuing the
copycluster
command again. The command copies only new data that has arrived after the initial copy.
Follow these steps:
  1. Log in to the source cluster with the database administrator account.
  2. Issue the following command:
    vbr.py --task copycluster --config-file
    CopyConfigurationFile
    .ini
The recent data in the source cluster is copied.
Verify the Copy of the Recent Data
To ensure the integrity of your data, verify the data of the source cluster that you copied.
Follow these steps:
  1. From any node in the cluster, open the Vertica SQL prompt by issuing the following command:
    /opt/vertica/bin/vsql -U dauser
  2. Verify the timestamp of these key database tables by running the following queries:
    SELECT to_timestamp(max(tstamp)) from dauser.reach_rate;
    SELECT to_timestamp(max(tstamp)) from dauser.ifstats_rate;
    The date and time must correspond to the time when you started the copy.
Update the Database Connection Information
Prepare to point the data aggregator to the target database. If you are not migrating the data aggregator, to enable communication between the data aggregator and the new data repository cluster, update the database connection information. If you are migrating the data aggregator, migrate the data aggregator.
For more information about how to migrate the data aggregator, see Migrate the Data Aggregator.
Follow these steps:
  1. Log in to the data aggregator host.
  2. Open the
    dbconnection.cfg
    file by issuing the following command:
    vi
    <installation_directory>
    /IMDataCollector/
    apache-karaf-*
    /etc/dbconnection.cfg
    • installation_directory
      The installation directory for the data aggregator.
      Default:
      /opt
    • apache-karaf-*
      The installation directory for Apache Karaf.
      Example:
      apache-karaf-4.2.6
  3. Update the following parameter with the hostnames of the new data repository cluster:
    dbHostNames=
    hostname1,hostname2,hostname3
The data repository migration is complete.
Restart the Data Aggregator
Follow these steps:
  1. Log in to the data aggregator host as the root user or a sudo user.
    If you installed the data aggregator as the sudo user, you set up a sudo command alias for the service
    dadaemon
    command. Use the sudo commands.
  2. Do one of the following steps:
    • Start the data aggregator service by issuing the following command:
      service dadaemon start
    • (Fault-tolerant environments) Enable the fault-tolerant data aggregator so that it can start when necessary by issuing one of the following commands:
      • RHEL 6.x:
        service dadaemon activate
      • RHEL 7.x/8.x, SLES, or OL:
        <installation_directory>
        /IMDataAggregator/scripts/dadaemon activate
        • installation_directory
          The installation directory for the data aggregator.
          Default:
          /opt
The data aggregator starts and synchronizes with
NetOps Portal
and the data repository. When the
iptables
entries are removed, the data collector sends any queued poll responses to the data aggregator.
If the queued data exceeds the disk space limit, then the data collector discards the oldest data. This can result in a data gap.
Monitor the Data Aggregator Restart Process
Follow these steps:
  1. After waiting for few minutes to let the data aggregator restart, log in to the data aggregator host and navigate to the
    /opt/IMDataAggregator/data/performance-spool
    directory.
  2. Verify that there are no data transfer object (DTO) files with a size greater than zero.
  3. Enable traffic from the Simple Network Management Protocol (SNMP) data collector with largest number of polled items by issuing the following command:
    iptables -D INPUT -s
    DC_IP
    -j DROP
    The data aggregator starts schema validation and processing of cached and new polled data from this data collector.
  4. After the data aggregator system utilization decreases, do the following:
    1. Enable traffic from the remaining SNMP data collectors.
    2. Enable traffic from the
      DX NetOps Mediation Manager
      data collectors.
Validate the Migration
Ensure that the system is healthy and then validate that the migrated data is present.
Follow these steps:
  1. Log in to
    NetOps Portal
  2. Verify that the system is healthy. Do the following steps:
    1. For the data aggregator, do the following:
      1. Navigate to the data aggregator data source list.
      2. Verify that the system status is good.
      3. Verify that the data aggregator data source is available.
      4. Verify the
        Last Polled On
        data and time.
    2. For the data collector, do the following:
      1. Navigate to the data collector list.
      2. Verify that the data collectors are up and collecting data.
    For more information:
  3. Validate that the migrated data is present. Go to the
    Infrastructure Overview
    dashboard, and then verify that data is available for the following time ranges:
    • Last hour
      The system is receiving SNMP poll responses when poll data exists for the last hour.
    • Last 7 days
      Data was properly migrated when poll data exists for the last seven days.
    For more information about this dashboard, see Out-of-the-Box Dashboards.