Upgrade the Data Repository

Upgrade the Data Repository first. If the dr_validate.sh script detects issues that you must resolve before the upgrade, you can restart your system while you resolve the issue.
capm310
Upgrade the Data Repository first. If the dr_validate.sh script detects issues that you must resolve before the upgrade, you can restart your system while you resolve the issue.
Red Hat Enterprise Linux (RHEL) 5.x is now deprecated. Data Repository upgrades on RHEL 5.x systems fail before the Vertica upgrade begins. To proceed with an upgrade, migrate the Data Repository to an RHEL 6.x system. For more information, see Migrate the Data Repository.
To upgrade the Data Repository, complete the following steps:
2
Verify the Prerequisites
Verify the following prerequisites before you upgrade Data Repository:
  • Verify whether the dialog package is installed on each Data Repository host:
    rpm -qa | grep ^dialog
    If the command does not return any results, install the dialog package:
    yum install dialog
    If this package is not installed, the validation and installation scripts fail.
  • Verify that the Data Repository host has at least 2 GB of swap space.
  • Verify that the Data Repository hosts use the ext4 or ext3 file system for data and catalog directories.
    The database performs best with the ext4 file system.
  • Verify that you are not using Logical Volume Manager (LVM) for data and catalog directories.
  • Verify the zip and unzip packages are installed. If these packages are not installed, use the following command to install them:
    yum -y install zip unzip
Verify Hardware and Network Performance
capm310
Vertica provides a set of utilities that test the performance of your hardware for the Data Repository. To verify that the environment is ideal for the database, run these tests before you upgrade the Data Repository.
3
3
If you have already installed the Data Repository, you can perform these tests at any time to verify performance. The utilities are available on each node in
/opt/vertica/bin
.
If the tests do not meet the recommendations, fix the issues before you continue the upgrade.
vcpuperf
This utility measures the CPU processing speed of the host and compares the speed against benchmarks for common server CPUs. The utility measures how long the server requires to complete the test, and determines whether CPU throttling is enabled.
Follow these steps:
  1. Execute the following command on
    each
    Data Repository node:
    ./vcpuperf > /tmp/vcpuperf.out
  2. Verify that the  performance meets the following requirements:
    • The CPU time is consistent with the benchmark values in the output.
    • The low load time and high load time are within 10 microseconds. If the difference is grater than 50 microseconds, CPU throttling might be enabled on your system. Disable CPU throttling.
Example:
The following example shows the return from this utility:
$ /opt/vertica/bin/vcpuperf
Compiled with: 4.1.2 20080704 (Red Hat 4.1.2-52)
Expected time on Core 2, 2.53GHz: ~9.5s
Expected time on Nehalem, 2.67GHz: ~9.0s
Expected time on Xeon 5670, 2.93GHz: ~8.0s
This machine's time:
CPU Time: 7.740000s
Real Time:7.740000s
Some machines automatically throttle the CPU to save power.
This test can be done in <100 microseconds (60-70 on Xeon 5670, 2.93GHz).
Low load times much larger than 100-200us or much larger than the corresponding high load time
indicate low-load throttling, which can adversely affect small query / concurrent performance.
This machine's high load time: 67 microseconds.
This machine's low load time: 64 microseconds.
This test was performed on a system with 2.67-GHz processors, so the real time is acceptable. The difference between the high load time and low load time is within the expected tolerance.
For more information about this utility, see the Vertica documentation.
vioperf
This utility tests the performance of the disk input and output (I/O). The utility performs a series of reads and writes.
Follow these steps:
  1. Execute the following commands on
    each
    Data Repository node:
    ./vioperf /
    data
    --duration=60sec > /tmp/vioperf.out
    /data
    is the full path of the data directory.
    ./vioperf /
    catalog
    --duration=60sec > /tmp/vioperf.out
    /catalog
    is the full path of the catalog directory.
  2. Verify the Write and Read counter values at least 40 MB/s per core.
    The recommended I/O is 40 MB/s per physical core on each node. For example, the recommended I/O rate for a node with 2 hyper-threaded six-core CPUs (12 physical cores) is 480 MB/s.
    If the
    thread count
    column shows a value of 1, the utility cannot determine the number of cores. Add the following arguement to the command to run the utility:
    --thread-count=
    CORES
    Cores
    defines the number of cores in the system as a fixed integer.
Example:
The following example shows the return from this utility for the data directory:
The minimum required I/O is 20 MB/s read and write per physical processor core on each node, in full duplex i.e. reading and writing at this rate simultaneously, concurrently on all nodes of the cluster. The recommended I/O is 40 MB/s per physical core on each node. For example, the I/O rate for a server node with 2 hyper-threaded six-core CPUs is 240 MB/s required minimum, 480 MB/s recommended.
Using direct io (buffer size=1048576, alignment=512) for directory "/drdata"
test      | directory                                       | counter name        | counter value       | counter value (10 sec avg)      | counter value/core  | counter value/core (10 sec avg) |
thread count
  | %CPU  | %IO Wait  | elapsed time (s)| remaining time (s)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Write
     | /drdata                                         | MB/s                | 873                 | 873                             |
54.5625 
            | 54.5625                         | 16            | 29    | 40        | 10              | 5               
Write
     | /drdata                                         | MB/s                | 868                 | 865                             |
54.25
               | 54.0625                         | 16            | 28    | 30        | 15              | 0               
ReWrite   | /drdata                                         | (MB-read+MB-write)/s| 275+275             | 275+275                         | 17.1875+17.1875     | 17.1875+17.1875                 | 16            | 13    | 21        | 10              | 5               
ReWrite   | /drdata                                         | (MB-read+MB-write)/s| 242+242             | 178+178                         | 15.125+15.125       | 11.125+11.125                   | 16            | 7     | 17        | 15              | 0               
Read
      | /drdata                                         | MB/s                | 735                 | 735                             |
45.9375
             | 45.9375                         | 16            | 11    | 23        | 10              | 5               
Read
      | /drdata                                         | MB/s                | 786                 | 786                             |
49.125 
             | 49.125                          | 16            | 26    | 25        | 15              | 0               
SkipRead  | /drdata                                         | seeks/s             | 4511                | 4511                            | 281.938             | 281.938                         | 16            | 14    | 19        | 10              | 5               
SkipRead  | /drdata                                         | seeks/s             | 4477                | 4407                            | 279.812             | 275.438                         | 16            | 3     | 15        | 15              | 0               
This server has 16 cores. The Read and Write counter values indicate the I/O is greater than 40 MB/s per core.
For more information about this utility, see the Vertica documentation.
vnetperf
This utility tests the network performance of the Data Repository hosts. The utility measures network latency and the throughput for the TCP and UDP protocols.
This utility causes a high network load and affects database performance. Do not run this utility while the database is running.
Follow these steps:
  1. Log in as a user that has passwordless ssh between the nodes.
  2. Execute the following command on
    one
    of the Data Repository nodes:
    ./vnetperf --hosts
    DAhost
    ,
    DRhost1
    ,
    DRhost2
    ,
    DRhost3
    > /tmp/vnetperf.out
    Specify the hostname or IP address of the Data Aggregator host and each Data Repository host.
  3. Verify that the network performance meets the following requirements:
    • Round-trip time (RTT) latency of 200 microseconds or less
    • Clock skew under 1 second
    • Throughput of 800 MB/s or more
      The utility runs a series of throttled tests. Verify the throughput for the highest speed test.
For more information about this utility, see the Vertica documentation.
Verify the Limit on the Number of Open Files
Verify that the user that is installing Data Repository has a value of at least 65536 on the number of open files. Set this value permanently. Complete this procedure for each node in the Data Repository cluster.
Follow these steps:
  1. As the root user or a sudo user, log in to the Data Repository host node.
  2. Run the following command:
    su dradmin
  3. Verify the number of open files:
    ulimit -n
    The command returns the ulimit number. This number must be at least 65536.
    The number must be the same on all nodes in the cluster.
  4. If this number is not at least 65536, do the following steps:
    1. Change the ulimit for the open files limit to 65536:
      ulimit -n 65536
    2. Open the /etc/security/limits.conf file on each Data Repository node, and add the following lines:
      # Added by Vertica
      * soft nofile 65536
      # Added by Vertica
      * hard nofile 65536
      # Added by Vertica
      *       soft    fsize   unlimited
      # Added by Vertica
      *       hard    fsize   unlimited
    3. Restart the sshd service on each Data Repository node:
      service sshd restart
      If you do not have the restart argument, stop and start sshd:
      service sshd stop
      service sshd start
    4.  Verify that the number of open files is set properly:
      ulimit -n
      The command returns the ulimit number that you specified earlier.
Remove mergeout_partitions.sh
If you used the mergeout_partitions.sh script, remove the cron that runs the script. The update to Vertica 7.1.2-6 removes the necessity for this script.
If you continue to run the mergeout_partitions.sh script, the Data Repository will suffer from degraded performance after the upgrade.
Disable the Automatic Recovery of the Data Aggregator Process
The recovery process restarts the Data Aggregator. Disable the automatic recovery before you upgrade.
Follow these steps:
  1. Log in to the Data Aggregator host as the root user.
  2. Open a console and type the following command:
    crontab  -e
    A vi session opens.
  3. If the following line exists, add # to the beginning of the line to comment it out:
    #
     * * * * * /etc/init.d/dadaemon start > /dev/null
    The automatic recovery is disabled.
Stop Data Collectors and Data Aggregator
Before the upgrade, stop the Data Collector and Data Aggregator processes.
Follow these steps:
  1. On each Data Collector host, open a command prompt, and type the following command:
    /etc/init.d/dcmd stop
  2. On the Data Aggregator host, open a command prompt, and type the following command:
    /etc/init.d/dadaemon stop
Upgrade Data Repository
 You can initiate the Data Repository installation from any of the hosts in the cluster. The required software components are pushed to the other nodes during the installation.
Follow these steps:
  1. Log in to one of the Data Repository nodes as the root user.
  2. Determine which hosts Vertica is running on:
    1. As the database administrator user, open the Vertica Administration Tools:
      /opt/vertica/bin/adminTools
    2. Select option 6 (Configuration Menu).
    3. Select option 3 (View Database).
    4. Select the database.
    5. Note the IP addresses and database name for use later in this procedure.
    6. Exit the adminTools utility, and revert to the root user or sudo user.
  3. Copy the installDR.bin file to /tmp.
  4. Change directory to the location of the installDR.bin file:
    cd /tmp
  5. Change the executable permissions:
    chmod u+x installDR.bin
  6. Extract the installation files:
    ./installDR.bin
  7. Follow the instructions in the console.
    This script extracts the following installation scripts:
    • dr_validate.sh
      verifies that the system meets the prerequisites for the Data Repository upgrade.
    • dr_install.sh
      upgrades the Vertica database.
  8. If available, copy the existing drinstall.properties to the following directory:
    /opt/CA/IMDataRepository_vertica7
    This
     
    path is the default location.
  9. Change directories to the location where you extracted the installation scripts:
    cd /opt/CA/IMDataRepository_vertica7
    This path is the default location.
  10. Verify that the parameters in the drinstall.properties file are correct. Review the following parameters:
    • DbAdminLinuxUser=
      The
      Linux user who is created to serve as the Vertica database administrato
      r
      Default
      : dradmin
    • DbAdminLinuxUserHome=
      The Vertica Linux database administrator user home directory
      Default:
      /export/dradmin
    • DbDataDir=
      The location of the data directory
      Default:
      /data
      To verify the data directory or catalog directory, complete the following steps:
      1. Open the /opt/vertica/config/admintools.conf file.
      2. Scroll down until you see the [Nodes] section.
      3. Locate one of the lines that begins with v_
        dbname
        _node
        ####
        .
        This line contains the IP address of the node, the location of the catalog directory, and the location of the data directory.
        Example:
        v_drdata_node0001 = 10.42.1.1,/catalog,/data
    • DbCatalogDir=
      The location of the catalog directory
      Default:
      /catalog
    • DbHostNames=
      The comma-delimited list of hostnames for Data Repository
      Default:
      yourhostname1,yourhostname2,yourhostname3
    • DbName=
      The database name
      Default:
      drdata
      This parameter is case-sensitive.
    • DbPwd=
      The database password
      Default:
      dbpass
    The InstallDestination is no longer used and can be safely removed.
  11. Verify that Data Repository is running.
  12. Run the validation script:
    ./dr_validate.sh -p drinstall.properties
    The script validates the system settings. Review and resolve any errors or warnings. You can run this script multiple times to verify that all system configuration options are set properly. The validation script may prompt you to reboot.
  13. Use Vertica adminTools to stop the Data Repository:
    1. Switch to the database admin user.
    2. Open adminTools:
      /opt/vertica/bin/adminTools
    3. Select (4) Stop Database.
    4. Select the database, confirm the selection, and provide the password.
    5. Exit the adminTools utility, and revert to the root user or sudo user.
  14. Run the installation script:
    ./dr_install.sh -p drinstall.properties
    The installation script upgrades the data repository and disables unnecessary Vertica processes. You might be prompted for the Vertica Linux database administrator user password.
    If passwordless SSH is not set up, the script prompts for the password to set up.
    If the installation script returns a WARN message for LVM on directories that Vertica does not use, contact CA Support.
  15. Verify that you upgraded Data Repository correctly by doing the following steps:
    1. As the database administrator user, open the Vertica Administration Tools:
      /opt/vertica/bin/adminTools
    2. Verify that the top of the banner indicates that the database version is 7.2.3-11.
  16. Restart the Data Repository selecting option 3 (Start Database) from the main menu of the Administration Tools dialog.
    If this is the first time you have stopped the database after the 30 June 2015 leap second, Vertica might fail to start. For more information, see Vertica Fails to Start.
    The Data Repository restarts.
Update the Backup
This release includes new functionality for the Data Repository backup. Previous backups and backup configuration files are not compatible with this release. Set up a new version of the backup. For more information, see Back Up the Data Repository.