Migrate the Data Repository

To migrate the Data Repository, install Vertica on the new cluster and migrate the data. The following situations might require migration:
capm370
To migrate the Data Repository, install Vertica on the new cluster and migrate the data. The following situations might require migration:
  • You are moving to new hosts for a major OS upgrade (for example, RHEL 6.9 to RHEL 7.3).
  • The current database hardware no longer meets sizing requirements.
  • You are moving from virtual machines to physical hardware for the database.
You must upgrade the existing system to the product version you are migrating to before migrating.
For more information about copying the database to another cluster, see the Vertica documentation.
2
Depending on the amount of data, migrating the Data Repository can take a significant amount of time.
To minimize downtime, perform incremental backups after your initial backup.
Prepare to Install the Destination Data Repository
Before you install Vertica on the new cluster, prepare the environment for the installation. For more information about preparing the destination cluster, see Prepare to Install the Data Repository.
The target cluster must have the following things:
  • The same number of nodes the source cluster
  • A database with the same name as the source database
    The target database can be empty.
  • The same node names as the source cluster
    The nodes names that are listed in the NODES system tables on both clusters must match. To change node names post install on the new system to match the existing one, contact Support.
  • Be accessible from the source cluster
  • The same database administrator account and all nodes must allow a database administrator of the source cluster to log in through SSH without a password
    Passwordless access within the cluster is not the same as passwordless access between clusters. The SSH ID of the administrator account on the source cluster and the target cluster are likely not the same. You must configure each host in the target cluster to accept the SSH authentication of the source cluster.
  • Adequate disk space for the
    vbr --task copycluster
    command to complete.
Install Vertica on the New Cluster
The installation process is the same as a normal Data Repository installation, for more information, see Install the Data Repository.
Use the same configuration for the new cluster as for the source cluster. For example, the vertica version, node count, database name, administrator, user, catalog directory, and data directory must be the same as the original Data Repository.
Migrate the Cluster
After you install the database on the new cluster, migrate the existing data. The migration uses the copy cluster command, which simultaneously backs up the existing database and restores the data to the new cluster. The copy cluster is configured and started and run on the source system to point to the new system. You can run copy cluster on a single node on the source system and it copies over to all nodes on the new system. Copy cluster copies all data in the Data Repository from before you run the command. Because
CA Performance Management
continues to collect data during the migration, the process requires multiple runs of the copy cluster command.
3
Create a Configuration File for Copy Cluster
Before you can configure the target cluster, you need to know the exact names that
admintools
supplied to all nodes in the source database.
To see the node names, run a query similar to following example:
VMart=> select node_name from nodes;
node_name
------------------
v_vmart_node0001
v_vmart_node0002
v_vmart_node0003
(3 rows)
The copy cluster command requires a configuration file that includes the necessary information. You can create the file in any desired location on the source system and give it any name with a
.ini
extension. In the configuration file, specify the host names of nodes in the target cluster as the backup hosts. Specify the nodes in a
[Mapping]
section as shown in the following example. Specify the full node name, not the short name (for example,
v_vmart_node0001
, not
node0001
).
Example:
The following example configuration file is set up to copy a database on a three node cluster (
v_vmart_node0001
,
v_vmart_node0002
, and
v_vmart_node0003
) to another cluster consisting of nodes: test-host01, test-host02, and test-host03:
The dbName parameter is case-sensitive.
[Misc]
snapshotName = CopyVmart
restorePointLimit = 5
objectRestoreMode = createOrReplace
tempDir = /tmp/vbr
retryCount = 5
retryDelay = 1
[Database]
dbName =
vmart
dbUser =
dradmin
dbPassword =
password
dbPromptForPassword = False
[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
[Mapping]
; backupDir is not used for cluster copy
v_vmart_node0001= test-host01
v_vmart_node0002= test-host02
v_vmart_node0003= test-host03
Ensure the
/tmp/vbr
directory has read and write permissions.
Stop the Target Database
Before you start the migration, shut down the database on the target cluster.
Follow these steps:
  1. Log in to the target database cluster as the database admin user.
  2. Open Vertica admin Tools:
    /opt/vertica/bin/adminTools
  3. Select (4) Stop Database. Wait for the shutdown to complete before you run copy cluster.
Copy Historical Data
After you install the database on the new cluster, copy the data from the existing database. The copy cluster command copies all information from before you initiate the command. New data continues to come in while the command is running, but the command does not copy this data. The target cluster must be stopped before you invoke copy cluster.
Follow these steps:
  1. Log in to the source cluster with the database administrator account.
  2. Ensure passwordless SSH is enabled for the database administrator account. In a cluster installation, ensure that passwordless ssh keys are set up for
    each
    host that is participating in the cluster.
    ssh
    hostname
    ls
    hostname
    is the name of the host where Data Repository is installed.
    If the passwordless ssh key is set up, you are
    not
    prompted for a password. If youare prompted for a password, set up passwordless ssh:
    1. Press Ctrl+C.
    2. Set up the Linux user account for the database administrator user with a passwordless ssh key:
      ssh-keygen -N "" -t rsa -f ~/.ssh/id_rsa
      cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys2
      chmod 644 ~/.ssh/authorized_keys2
    3. Copy the SSH credentials to the other hosts in the cluster:
      ssh-copy-id -i
      dradmin
      @
      other-hostname-in-the-cluster
    4. Confirm that you are
      not
      prompted for a password:
      ssh hostname ls
  3. Run the copy cluster command:
    vbr.py --task copycluster --config-file
    CopyClusterConfigurationFile
    .ini
    The command copies the historical data for the database and displays the following message:
    > vbr.py --config-file CopyVmart.ini --task copycluster
    Preparing...
    Copying...
    1871652633 out of 1871652633, 100%
    All child processes terminated successfully.
    copycluster done!
    If the
    copycluster
    command fails, ensure passwordless SSH is enabled for the database administrator account.
Verify the Copy of the Historical Data
After the copy cluster process completes, ensure the integrity of your data.
Follow these steps:
  1. Log in to the target database cluster as the database admin user.
  2. Open Vertica adminTools:
    /opt/vertica/bin/adminTools
  3. Start the database.
  4. From any node in the cluster, open open the Vertica SQL prompt:
    /opt/vertica/bin/vsql -U dauser
  5. Run the following queries to verify the timestamp of these key database tables:
    SELECT to_timestamp(max(tstamp)) from dauser.reach_rate;
    SELECT to_timestamp(max(tstamp)) from dauser.ifstats_rate;
    The date and time must correspond to the time when you started the copy.
  6. Open adminTools, and stop the database.
Stop the Data Aggregator
To maintain integrity of your data, stop the Data Aggregator before the final copy of the Data Repository. Polling continues on Data Collector if it is running and polling when Data Aggregator is stopped. Data Collector queues polled data for future delivery to Data Aggregator.
Follow these steps:
  1. Log in to the Data Aggregator host as the root user or a sudo user.
    If you installed Data Aggregator as the sudo user, you set up a sudo command alias for the service dadaemon command. Use the sudo commands.
  2. Use firewall rules to block traffic from all the Data Collectors. Run the following command for each Data Collector:
    iptables -A INPUT -s
    DC_IP
    -j DROP
    DC_IP
    Specifies the IP of the Data Collector.
  3. Do one of the following steps:
    • Stop the Data Aggregator service:
      service dadaemon stop
      For RHEL 7.x or OL,
      service
      invokes
      systemctl
      . You can use
      systemctl
      instead.
    • (Fault tolerant environment) If the local Data Aggregator is running, run one the following commands to shut it down and prevent it from restarting until maintenance is complete:
      • RHEL 6.x:
        service dadaemon maintenance
      • RHEL 7.x, SLES, or OL:
        DA_Install_Directory/scripts/dadaemon maintenance
    The Data Aggregator stops.
Copy Recent Data
After you stop the Data Aggregator, run the copy cluster command again to copy recent data. The Data Repository copies only new data that arrive after the initial copy.
Follow these steps:
  1. Log in to the source cluster with the database administrator account.
  2. Run the copy cluster command:
    vbr.py --task copycluster --config-file
    CopyConfigurationFile
    .ini
    The command copies the recent data.
Verify the Copy of the Recent Data
To ensure the integrity of your data, verify the data.
Follow these steps:
  1. Log in to the target database cluster as the database admin user.
  2. Open Vertica admin Tools:
    /opt/vertica/bin/adminTools
  3. Start the database.
  4. From any node in the cluster, open the Vertica SQL prompt:
    /opt/vertica/bin/vsql -U dauser
  5. Run the following queries to verify the timestamp of these key database tables:
    SELECT to_timestamp(max(tstamp)) from dauser.reach_rate;
    SELECT to_timestamp(max(tstamp)) from dauser.ifstats_rate;
    The date and time must correspond to the time when you started the copy.
Update the Database Connection Information
If you are not migrating the Data Aggregator, to enable communication between the Data Aggregator and the new Data Repository cluster, update the database connection information.
Follow these steps:
  1. Log in to the Data Aggregator host.
  2. Open the following file:
    vi /opt/IMDataAggregator/apache-karaf-
    version
    /etc/dbconnection.cfg
  3. Update the following parameter with the hostnames of the new Data Repository cluster:
    dbHostNames=
    hostname1,hostname2,hostname3
Restart Data Aggregator
After the migration is complete, restart the Data Aggregator.
Follow these steps:
  1. Log in to the Data Aggregator host as the root user or a sudo user.
    If you installed Data Aggregator as the sudo user, you set up a sudo command alias for the service dadaemon command. Use the sudo commands.
  2. Do one of the following steps:
    • Start the Data Aggregator service:
      service dadaemon start
    • (Fault tolerant environment) Run one the following commands to enable the fault tolerant Data Aggregator so that it can start when necessary:
      • RHEL 6.x:
        service dadaemon activate
      • RHEL 7.x, SLES, or OL:
        DA_Install_Directory
        /scripts/dadaemon activate
    Data Aggregator starts and synchronizes with CA
    CA Performance Center
    and the Data Repository. When the iptables entries are removed, any queued, polled data on the Data Collectors is sent to the Data Aggregator. The oldest data is discarded if the queued data exceeds a disk space limit that is configured on the Data Collector system. As a result, there is a gap in the polled reporting data.
  3. Monitor the Data Aggregator restart process:
    1. Log in to the Data Aggregator host and navigate to the following directory:
      /opt/IMDataAggregator/performance-spool
    2. Wait for few minutes after starting the Data Aggregator service. Verify that no DTO files exist with a size greater than zero.
    3. Enable traffic from the SNMP Data Collector with largest number of polled items:
      iptables -D INPUT -s
      DC_IP
      -j DROP
      Data Aggregator starts schema validation and processing of cached and new polled data from this Data Collector.
    4. After the Data Aggregator system utilization decreases, enable traffic from the remaining SNMP Data Collectors.
    5. After the Data Aggregator system utilization decreases, enable traffic from the CAMM Data Collectors.
Verify the Migration
After the Data Aggregator startup is complete, log in to
CA Performance Center
, and verify the following indicators:
  • The system status is good and the Data Aggregator data source is available.
  • Verify the Last Polled on data and time.
  • Navigate to the Data Collector List and verify that all Data Collectors are up and collecting data.
  • Open the Infrastructure Overview dashboard, and verify that data is available for the following time ranges:
    • Last hour
    • Last 7 days