Upgrade the Data Repository

Upgrade the data repository first. If the dr_validate.sh script detects issues that you must resolve before the upgrade, you can restart your system while you resolve the issue.
Upgrade the data repository first.
The Vertica release 9.1.1-5 is the same for
DX NetOps Performance Management
3.7 and 20.2. Therefore, a data repository upgrade is unnecessary from 3.7 to this release.
Use the following workflow to upgrade the data repository:
2
Verify the Prerequisites
Verify the following prerequisites before you upgrade the data repository:
  • Verify whether the dialog, chrony, zip, and unzip packages are installed on each data repository host:
    The chrony package is required only for RHEL 7.x and OL 7.x.
    rpm -qa | grep ^dialog
    rpm -qa | grep ^chrony
    rpm -qa | grep ^zip
    rpm -qa | grep ^unzip
    If a required package for your OS is missing, install the package:
    If you are not the root user, use the sudo prefix.
    yum -y install dialog
    yum -y install chrony
    yum -y install zip
    yum -y install unzip
    SLES:
    zypper install dialog -y
    zypper install zip -y
    zypper install unzip -y
    If this package is not installed, the validation and installation scripts fail.
  • Verify that the data repository host has at least 2 GB of swap space.
  • Verify that the data repository hosts use the ext4 file system for data and catalog directories. If not, remake the file system as ext4.
    The default file system for RHEL 7.x and OL 7.x is the XFS file system. The default file system for SLES is btrfs. Vertica does not support XFS or btrfs. The database supports only the ext4 file system.
  • Verify that you are not using Logical Volume Manager (LVM) for data and catalog directories.
Verify Hardware and Network Performance
You can verify that the environment is ideal for the database by testing the performance of your hardware for the data repository. Run the following Vertica utilities
before
you upgrade the data repository:
3
3
Run these utilities at any time. They are available on each node in the
/opt/vertica/bin
directory.
If the tests do not meet the recommendations, fix the issues before you continue the upgrade.
vcpuperf
The
vcpuperf
utility measures the CPU processing speed of the host and compares the speed against benchmarks for common server CPUs. The utility measures how long the server requires to complete the test, and determines whether CPU throttling is enabled.
For more information about this utility, see the Vertica documentation.
Follow these steps:
  1. Issue the following command on
    each
    data repository node:
    ./vcpuperf > /tmp/vcpuperf.out
  2. Verify that the  performance meets the following requirements:
    • The CPU time is consistent with the benchmark values in the output.
    • The low load time and high load time are within 10 microseconds. If the difference is grater than 50 microseconds, CPU throttling might be enabled on your system. Disable CPU throttling.
Example:
The following example shows the return from this utility:
$ /opt/vertica/bin/vcpuperf
Compiled with: 4.1.2 20080704 (Red Hat 4.1.2-52)
Expected time on Core 2, 2.53GHz: ~9.5s
Expected time on Nehalem, 2.67GHz: ~9.0s
Expected time on Xeon 5670, 2.93GHz: ~8.0s
This machine's time:
CPU Time: 7.740000s
Real Time:7.740000s
Some machines automatically throttle the CPU to save power.
This test can be done in <100 microseconds (60-70 on Xeon 5670, 2.93GHz).
Low load times much larger than 100-200us or much larger than the corresponding high load time
indicate low-load throttling, which can adversely affect small query / concurrent performance.
This machine's high load time: 67 microseconds.
This machine's low load time: 64 microseconds.
This test was performed on a system with 2.67-GHz processors, so the real time is acceptable. The difference between the high load time and low load time is within the expected tolerance.
vioperf
The
vioperf
utility tests the performance of the disk input and output (I/O). The utility performs a series of reads and writes.
For more information about this utility, see the Vertica documentation.
Follow these steps:
  1. Issue the following commands on
    each
    data repository node:
    ./vioperf /
    data
    --duration=60sec > /tmp/vioperf.out
    /data
    is the full path of the data directory.
    ./vioperf /
    catalog
    --duration=60sec > /tmp/vioperf.out
    /catalog
    is the full path of the catalog directory.
  2. Verify the Write and Read counter values at least 40 MB/s per core.
    The recommended I/O is 40 MB/s per physical core on each node. For example, the recommended I/O rate for a node with 2 hyper-threaded six-core CPUs (12 physical cores) is 480 MB/s.
    If the
    thread count
    column shows a value of 1, the utility cannot determine the number of cores. Add the following argument to the command to run the utility:
    --thread-count=
    CORES
    Cores
    defines the number of cores in the system as a fixed integer.
Example:
The following example shows the return from this utility for the data directory:
The minimum required I/O is 20 MB/s read and write per physical processor core on each node, in full duplex i.e. reading and writing at this rate simultaneously, concurrently on all nodes of the cluster. The recommended I/O is 40 MB/s per physical core on each node. For example, the I/O rate for a server node with 2 hyper-threaded six-core CPUs is 240 MB/s required minimum, 480 MB/s recommended.
Using direct io (buffer size=1048576, alignment=512) for directory "/drdata"
test | directory | counter name | counter value | counter value (10 sec avg) | counter value/core | counter value/core (10 sec avg) |
thread count
| %CPU | %IO Wait | elapsed time (s)| remaining time (s)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Write
| /drdata | MB/s | 873 | 873 |
54.5625
| 54.5625 | 16 | 29 | 40 | 10 | 5
Write
| /drdata | MB/s | 868 | 865 |
54.25
| 54.0625 | 16 | 28 | 30 | 15 | 0
ReWrite | /drdata | (MB-read+MB-write)/s| 275+275 | 275+275 | 17.1875+17.1875 | 17.1875+17.1875 | 16 | 13 | 21 | 10 | 5
ReWrite | /drdata | (MB-read+MB-write)/s| 242+242 | 178+178 | 15.125+15.125 | 11.125+11.125 | 16 | 7 | 17 | 15 | 0
Read
| /drdata | MB/s | 735 | 735 |
45.9375
| 45.9375 | 16 | 11 | 23 | 10 | 5
Read
| /drdata | MB/s | 786 | 786 |
49.125
| 49.125 | 16 | 26 | 25 | 15 | 0
SkipRead | /drdata | seeks/s | 4511 | 4511 | 281.938 | 281.938 | 16 | 14 | 19 | 10 | 5
SkipRead | /drdata | seeks/s | 4477 | 4407 | 279.812 | 275.438 | 16 | 3 | 15 | 15 | 0
This server has 16 cores. The Read and Write counter values indicate the I/O is greater than 40 MB/s per core.
vnetperf
The
vnetper
utility tests the network performance of the data repository hosts. The utility measures network latency and the throughput for the TCP and UDP protocols.
This utility causes a high network load and affects database performance. Do not run this utility while the database is running.
For more information about this utility, see the Vertica documentation.
Follow these steps:
  1. Log in as a user that has passwordless ssh between the nodes.
  2. Issue the following command on
    one
    of the data repository nodes:
    ./vnetperf --hosts
    DAhost
    ,
    DRhost1
    ,
    DRhost2
    ,
    DRhost3
    > /tmp/vnetperf.out
    Specify the hostname or IP address of the data aggregator host and each data repository host.
  3. Verify that the network performance meets the following requirements:
    • Round-trip time (RTT) latency of 200 microseconds or less
    • Clock skew under 1 second
    • Throughput of 800 MB/s or more
      The utility runs a series of throttled tests. Verify the throughput for the highest speed test.
Verify the Limit on the Number of Open Files
Verify that the user that is installing data repository has a value of at least 65536 on the number of open files. Set this value permanently. Complete this procedure for each node in the data repository cluster.
Follow these steps:
  1. As the root or sudo user, log in to the data repository host node.
  2. Issue the following command:
    su dradmin
  3. Verify the number of open files:
    ulimit -n
    The command returns the ulimit number. This number must be at least 65536.
    The number must be the same on all nodes in the cluster.
  4. If this number is not at least 65536, do the following steps:
    1. Change the ulimit for the open files limit to 65536:
      ulimit -n 65536
    2. Open the
      /etc/security/limits.conf
      file on each data repository node, and add the following lines:
      # Added by Vertica
      * soft nofile 65536
      # Added by Vertica
      * hard nofile 65536
      # Added by Vertica
      * soft fsize unlimited
      # Added by Vertica
      * hard fsize unlimited
    3. Restart the sshd service on each data repository node:
      service sshd restart
      For RHEL 7.x, and OL 7.x
      service
      invokes
      systemctl
      . You can use
      systemctl
      instead.
      If you do not have the restart argument, stop and start sshd:
      service sshd stop
      service sshd start
    4. Verify that the number of open files is set properly:
      ulimit -n
      The command returns the ulimit number that you specified earlier.
Disable the Automatic Recovery of the Data Aggregator Process
The recovery process restarts the data aggregator. Disable the automatic recovery before you upgrade.
For a fault-tolerant environment, put the inactive data aggregator and then the active data aggregator into maintenance mode.
Check to ensure that you know which data aggregator is active and which is inactive.
For more information about how to check this, see View System Status.
Follow these steps:
  1. Log in to the data aggregator host as the root user.
  2. Open a console and issue the following command:
    crontab -e
    A vi session opens.
  3. If the following line exists, add # to the beginning of the line to comment it out:
    #
    * * * * * /etc/init.d/dadaemon start > /dev/null
    The automatic recovery is disabled.
(Fault-tolerant Environment) Put a Data Aggregator into Maintenance Mode
If you are upgrading fault-tolerant data aggregators, put the data aggregator into maintenance mode before upgrading.
Check to ensure that you know which data aggregator is active and which is inactive.
For more information about how to check this, see View System Status.
Put the inactive data aggregator into maintenance mode first, then put the active data aggregator into maintenance mode.
For more information, see Upgrade Fault-Tolerant Data Aggregators.
Follow these steps:
  1. Log in to the data aggregator host as the root user or a sudo user.
  2. Issue
    one
    of the following commands to shut down the inactive data aggregator and prevent it from restarting until the upgrade is complete:
    • RHEL 6.x:
      service dadaemon maintenance
    • RHEL 7.x, SLES, or OL:
      DA_Install_Directory
      /scripts/dadaemon maintenance
Upgrade the Data Repository
You can initiate the data repository installation from any of the hosts in the cluster. The required software components are pushed to the other nodes during the installation.
Follow these steps:
  1. Log in to one of the data repository nodes as the root user.
  2. Determine on which hosts Vertica is running:
    1. As the database administrator user, open the Vertica Administration Tools:
      /opt/vertica/bin/adminTools
    2. Select option
      6 (Configuration Menu)
      .
    3. Select option
      3 (View Database)
      .
    4. Select the database.
    5. Note the IP addresses and database name for use later in this procedure.
    6. Exit the adminTools utility, and revert to the root user or sudo user.
  3. Copy the
    installDR.bin
    file to
    /tmp
    .
  4. Change directory to the location of the installDR.bin file:
    cd /tmp
  5. Change the executable permissions:
    chmod u+x installDR.bin
  6. Extract the installation files:
    ./installDR.bin
  7. Follow the instructions in the console.
    This script extracts the following installation scripts:
    • dr_validate.sh
      Verifies that the system meets the prerequisites for the data repository upgrade. If this script detects issues that you must resolve before the upgrade, you can restart your system while you resolve the issue.
    • dr_install.sh
      Upgrades the Vertica database.
  8. If available from the data repository with the older version of Vertica, copy the existing
    drinstall.properties
    to the following directory:
    /opt/CA/IMDataRepository_vertica
    Version
    This path is the default location.
  9. Change directories to the location where you extracted the installation scripts:
    cd /opt/CA/IMDataRepository_vertica
    Version
    This path is the default location.
  10. Verify that the parameters in the
    drinstall.properties
    file are correct. Review the following parameters:
    • DbAdminLinuxUser=
      The Linux user who is created to serve as the Vertica database administrator
      Default
      : dradmin
    • DbAdminLinuxUserHome=
      The Vertica Linux database administrator user home directory
      Default:
      /export/dradmin
    • DbDataDir=
      The location of the data directory
      Default:
      /data
      To verify the data directory or catalog directory, complete the following steps:
      1. Open the
        /opt/vertica/config/admintools.conf
        file.
      2. Scroll down until you see the [Nodes] section.
      3. Locate one of the lines that begins with v_
        dbname
        _node
        ####
        .
        This line contains the IP address of the node, the location of the catalog directory, and the location of the data directory.
        Example:
        v_drdata_node0001 = 10.42.1.1,/catalog,/data
    • DbCatalogDir=
      The location of the catalog directory
      Default:
      /catalog
    • DbHostNames=
      The comma-delimited list of hostnames for data repository
      Default:
      yourhostname1,yourhostname2,yourhostname3
    • DbName=
      The database name
      Default:
      drdata
      This parameter is case-sensitive.
    • DbPwd=
      The database password
      Default:
      dbpass
      You can use special characters (except for single quotation marks) in passwords. If the
      DbPwd
      property is not found or blank, the script prompts for this information at runtime.
    The InstallDestination is not used. You can safely remove it.
  11. Verify that the data repository is running. On the data repository host, issue the following command:
    /opt/vertica/bin/vsql -U
    dauser
    -w
    dapass
    -c 'select version()'
    The following response is expected:
    version
    ------------------------------------
    Vertica Analytic Database vx.x.x-x
  12. On the data aggregator host, open a command prompt. Do
    one
    of the following steps:
    • Stop the Data Aggregator service:
      service dadaemon stop
    • (Fault-tolerant environments) If the local data aggregator is running, run one the following commands to shut it down and prevent it from restarting until maintenance is complete:
      • RHEL 6.x:
        service dadaemon maintenance
      • RHEL 7.x, SLES, or OL:
        DA_Install_Directory
        /scripts/dadaemon maintenance
  13. Run the validation script:
    ./dr_validate.sh -p drinstall.properties
    You can use the
    -l
    flag to allow
    localhost
    as the value for the
    DbHostNames
    property. You can use the
    -n
    flag to skip database connectivity checks.
    The script validates the system settings. Review and resolve any errors or warnings. You can run this script multiple times to verify that all system configuration options are set properly. The validation script may prompt you to reboot.
  14. Use Vertica adminTools to stop the data repository:
    1. Switch to the database admin user.
    2. Open adminTools:
      /opt/vertica/bin/adminTools
    3. Select (4) Stop Database.
    4. Select the database, confirm the selection, and provide the password.
    5. Exit the adminTools utility, and revert to the root user or sudo user.
  15. Run the installation script:
    ./dr_install.sh -p drinstall.properties
    The installation script upgrades the data repository and disables unnecessary Vertica processes. You might be prompted for the Vertica Linux database administrator user password.
    If your username is not set up with passwordless SSH, the script prompts for the password to set up.
    If the installation script returns a WARN message for LVM on directories that Vertica does not use, contact CA Support.
  16. Verify that you upgraded the data repository correctly by doing the following steps:
    1. As the database administrator user, open the Vertica Administration Tools:
      /opt/vertica/bin/adminTools
    2. Verify that the top of the banner indicates that the database version is 9.1.1-5.
  17. Restart the data repository selecting option 3 (Start Database) from the main menu of the
    Administration Tools
    dialog.
    If this is the first time that you have stopped the database after the 30 June 2015 leap second, Vertica might fail to start.
    For more information, see Vertica Fails to Start.
    The data repository restarts.
Next Steps
Set up a new version of the data repository backup.
For more information, see Back Up the Data Repository.