Replace a Failed Gateway Database Node

This topic describes how to replace a failed database node in a gateway cluster. There are various node replacement scenarios based on your current environment configuration. The configuration applies to both Hardware and Virtual Appliance Gateways. Where there are differences between the two form factors, the instructions show "(Hardware only)" or "(Hardware and Virtual)".
gateway83
This topic describes how to replace a failed database node in a
Layer7 API Gateway
cluster. There are various node replacement scenarios based on your current environment configuration. The configuration applies to both Hardware and Virtual Appliance Gateways. Where there are differences between the two form factors, the instructions show "(Hardware only)" or "(Hardware and Virtual)".
This topic is intended for system administrators or other advanced technical users.
Prerequisites:
  • You have a recent backup image from which to restore a Gateway.
  • You have the license file for your Gateway.
Unless your Gateway is standalone (that is, a "cluster of one"), there is no down time involved in the replacement.
Contents:
Step 1: Prepare the Appliance
  1. (Hardware only)
    Inspect the replacement Gateway appliance to ensure that there is no damage during transport.
  2. (Hardware only)
    Rack-mount the appliance. Connect the power and network cables.
  3. (Hardware and Virtual)
    Ensure that the replacement appliance is running the same Software/Platform version. To check, run this command from a privileged shell:
    # rpm -qa | grep ssg
    For more information on starting up a new Virtual Appliance, download the
    CA API Gateway Virtual Appliance Getting Started
    from "CA API Management Technical Documentation" in Release Notes.
    Keep the privileged shell open for the next step.
Step 2: Stop Replication
Skip this step if you have a stand-alone Gateway, as there is no replication involved.
Run the following commands from the privileged shell to stop replication.
  1. On the Database Node, stop the MySQL slave process with this command:
    # mysql –e 'stop slave'
  2. On the Database Node, adjust the firewall with this command:
    # iptables –I OUTPUT –p tcp –-dport 3306 –j REJECT
  3. On the
    failed Database Node only
    , run this command:
    # iptables –I INPUT –i eth0 –p tcp –m tcp –s
    <IP_of_Database_Node>
    –-dport 3306 –j REJECT
Step 3: Configure the Network
  1. Connect to the Gateway appliance either through a KVM switch, Serial Console (Hardware Appliance) or vSphere Console (Virtual Appliance).
  2. Log in as the
    ssgconfig
    user. You are prompted to change your password.
    Note:
    The new password must conform to "Password Rules" in Troubleshooting Password Issues.
  3. In the Gateway main menu, select option
    1
    (Configure System Settings) and then option
    1
    (Configure networking and system time settings).
  4. Complete the prompts onscreen to configure the General Network Settings, Name Servers, and Time Zone. For more information, see Option 1 - Configure Networking and System Time Settings.
Step 4: Stand-alone Gateways: Create a New Database
If you have a stand-alone Gateway, create a new database as follows:
  1. Return to the Gateway main menu.
  2. Select option 2 (Display CA API Gateway configuration menu) and then option 2 (Create a new CA API Gateway database). Complete the prompts onscreen. For more information, see Gateway Configuration Menu (Appliance).
  3. Proceed to Step 5.
If your Gateway cluster includes a primary and secondary database node, proceed to the appropriate step:
  • If replacing a primary database node, see Step 6.
  • If replacing a secondary database node, see Step 7.
Step 5: Replacing the Database on a Stand-alone Gateway
  1. Copy the backup image file to this directory:
    /opt/SecureSpan/Gateway/config/backup/images/
  2. Restore the image with this command:
    # ssgrestore.sh -image -dbu –dbp
    For more information about the
    ssgrestore
    command, see Restore Gateways.
Step 6: Replacing a Primary Database Node
  1. On the new
    Primary
    Database Node,
    run this command:
    # /opt/SecureSpan/Appliance/bin/add_slave_user.sh
    Notes:
    • Enter the hostname or IP of the Secondary Database Node.
    • When prompted “Is this the Primary (1) or Secondary (2) database node?”, enter
      1
      .
  2. On the new
    Primary
    Database Node,
    run this command:
    # /opt/SecureSpan/Appliance/bin/create_slave.sh
    Note:
    • When prompted to clone the database, enter
      yes.
    • When warned about dropping the database, enter
      y
      .
       
  3. On the new
    Primary
    Database Node,
    run this command:
    # iptables –D INPUT –i eth0 –p tcp –m tcp –s
    <IP_of_Database_Node>
    –-dport 3306 –j REJECT
  4. On the
    Secondary
    Database Node,
    run these commands:
    # iptables –D OUTPUT –p tcp –-dport 3306 –j REJECT
    # /opt/SecureSpan/Appliance/bin/create_slave.sh
    When prompted to clone the database, enter
    no
    .
  5. Verify that replication has started by running the following command on each node:
    # mysql;show slave status\G;
    In the list that is displayed, look for the following lines:
    Slave_IO_Running: Yes
    Slave_SQL_Running: Yes
Step 7: Replacing a Secondary Database Node
  1. On the new
    Secondary
    Database Node,
    run this command:
    #/opt/SecureSpan/Appliance/bin/add_slave_user.sh
    Notes:
    • Enter the hostname or IP of the Primary Database Node.
    • When prompted “Is this the Primary (1) or Secondary (2) database node?”, enter
      2
      .
  2. On the new
    Secondary
    Database Node,
    run this command:
    # /opt/SecureSpan/Appliance/bin/create_slave.sh
    Notes:
    • When prompted to clone the database, enter
      yes
      .
    • When warned about dropping the database, enter
      y
      .
  3. On the new
    Secondary
    Database Node,
    run this command:
    # iptables –D INPUT –i eth0 –p tcp –m tcp –s
    <IP_of_Database_Node>
    –-dport 3306 –j REJECT
  4. On the
    Primary
    Database Node,
    run these commands:
    # iptables –D OUTPUT –p tcp –-dport 3306 –j REJECT
    # /opt/SecureSpan/Appliance/bin/create_slave.sh
    When prompted to clone the database, enter
    no.
     
  5. Verify that replication has started by running the following command on each node:
    # mysql;show slave status\G;
    In the list that is displayed, look for the following lines:
    Slave_IO_Running: Yes
    Slave_SQL_Running: Yes
Step 8: Post Configuration
  1. Start the Policy Manager and install the license. For more information, see "Install a License File" in Manage Gateway Licenses.
  2. Verify that the Gateway has started correctly by pinging the replacement node. For more information, see Ping URI Test.