Configure Management Center Failover

Management Center supports failover using two physical appliances. One appliance is delegated as the
primary
and the other as the
secondary
. After failover is configured, the secondary replicates data from the primary appliance. During continuous replication, users can perform all normal operations on the primary failover partner. Users cannot access the secondary failover partner—its sole purpose is to replicate actions occurring on the primary node so that it can take over if something happens to primary node.
Because the secondary failover partner replicates the primary partner's data, it is ready to take over if the primary failover partner becomes unresponsive.
Management Center supports three failover methods for your virtual appliance.
  • The vMotion feature of ESX. This method is recommended. If you have vMotion in your installation,
    Symantec
    recommends that you use it instead of the manual failover support described in this topic. See Verify VMware Requirements for more information.
  • Manual failover mode. If you choose this method, you must manually log into the secondary failover partner and make it the primary failover partner.
  • Automatic failover mode. If you choose this method, failover automatically occurs after the timeout value (set during failover configuration) expires. When using this method, you can optionally assign a virtual IP address to the primary failover partner and then use that virtual IP address as the secondary failover partner's primary IP addresss. This enables the secondary to automatically begin accepting request on that virtual IP-address when the primary failover partner is unresponsive. This method is supported for AWS and non-AWS deployments.
.

Important Failover Notes

  • A one-time authorization token is required to set up failover in
    Management Center
    2.x and later. The token is generated during the configuration of the primary partner and is good for 24 hours. See Configure Failover.
  • To ensure that the secondary failover partner has enough resources to replicate, and when required, take over the primary partner's function, the secondary partner must be sized appropriately. The secondary partner should have the same (or greater) disk size, CPUs, and memory.
  • For systems setup in failover, the data encryption key is kept in sync between the primary and secondary devices.
  • Management Center
    support multiple network interfaces.
    Symantec
    recommends that failover partners communicate over a separate channel.
  • You can use IPv6 for failover communication In
    Management Center
    2.x and later.
  • If you intend to upgrade the failover pair, you must first disable failover. After upgrade, you can then reestablish failover.
  • A
    Management Center
    assigned as the secondary partner can only be accessed by users logging in with the admin account. For example, to make the secondary partner the primary, you must be logged in with the admin account. See Add Local Users for more information.

Data Replicated Between Failover Partners

The following data is replicated on the failover partner:
  • Device data stored in the database.
  • Files in the
    Management Center
    file store
  • Policy and scripts (along with historical versions)
  • Device backups
  • PDM data from
    ProxySG
    appliances
  • Data protection key
  • Trusted certificates for servers; root CA installed by a user
  • The following configuration settings in
    Administration > Settings
    :
    • General
    • SMTP Alerts
    • SNMP Alerts
      Housekeeping
    • Mail Settings
    • Consent Banner
    • Hardware Monitor Settings
  • Optional replication:
    The user can also choose to replicate the following on the secondary failover partner:
    • Authentication (Active Directory/LDAP/RADIUS) configuration
    • Logging and alert configuration
    • Access Control List (ACL) configuration
    See failover for more information.
The following data is not replicated on the failover partner:
  • Management Center
    licensing and system settings
  • Management Center
    backup images stored on the device itself

Failover Configuration Limitations

During replication, configuration for both the primary and secondary failover partners is limited. Replication requires that both the primary and secondary partners run the same version of Management Center. To enforce this, the
installed-systems
CLI command is disabled on both failover partners (to deny installing and changing system images). If, for any reason, the system images do not match on the primary and secondary partners – replication is paused until the problems are resolved.
The secondary failover partner has stricter restrictions on what can be configured. In addition to not being able to manage system images, the following CLI commands are disabled on the secondary partner:
backup
(all commands)
license
(all commands)
http-proxy
(all commands)
service db-maintenance
service purge-vpm-cache
snmp
(all commands)
statistics-monitoring
(all commands)

Device Limitations

Because Web Security Service (WSS) devices are initially registered through a connection established only with the primary partner (which subsequently discards the credentials), WSS connections will fail if an event causes a failover to the secondary partner. In that event, you must re-authenticate to those WSS devices (
Network > Edit Device > Connection Parameters
).

General Failover Prerequisites

To prepare for failover:
  • Determine your failover mode: automatic or manual.
  • Ensure that port 2025 is open between the primary and secondary partners.
    Management Center
    failover employs an SSH connection.
  • Ensure DNS is working properly. SSH connections will attempt RDNS; you must have a valid DNS setup for the pair when using failover.
  • Ensure you have a method for recording the one-time authorization token generated while configuring the primary appliance. This token is required for configuring the secondary appliance.

Manual Failover Prerequisites

Meet the General Failover Prerequisites above and:
  • Identify a Management Center appliance to act as the primary failover partner. Record the IP address and password of the "admin" account of this device.
  • Identify a Management Center appliance to act as the secondary failover partner. Record the IP address and password of the "admin" account of this device.

Automatic Failover Prerequisites

Meet the General Failover Prerequisites above and:
  • Obtain an unused IP address within the primary failover partner's subnet. You will use this IP address to create a virtual IP address on the primary failover partner. The same virtual IP address will be the secondary failover partner's primary IP address.
  • AWS: If you are deploying automatic failover within AWS, the AWS role attached to your instance must allow you to assign or unassign private IP addresses. Otherwise, the virtual IP address will not switchover during a failover event.

Configure Failover

You must enable failover using the CLI.
Step 1—Configure the Primary Appliance
  1. Use an SSH client to log into the CLI of the Management Center appliance that is to be the primary failover partner.
  2. Enter Enable mode:
    #
    enable
  3. Enter Configuration mode:
    #
    configure
  4. Confirm that failover has not already been configured on the appliance:
    (config)#
    failover view
    Failover:
    Status: Disabled
  5. Do one of the following:
    • For manual failover, skip this step and go to step 6.
    • To enable automatic failover, you must assign a virtual IP address to the primary failover partner. This IP address can be any available IP address in the primary failover partner's subnet:
      (config)# virtual-ip address
      ip_address
      For example:
      (config)# virtual-ip address 192.0.2.24 Virtual-ip: Status: running, this node has virtual-ip Address: 192.0.2.24 Interface: 0:0 Peer: not defined
  6. Make this appliance the primary failover partner. This process generates a one-time authentication to be used for configuring the secondary partner.
    (config)#
    failover make-primary
    The command output is similar to the following:
    One-time initial authentication token for secondary node: 58f1ddaa6f878f96 Failover: Status: ERROR: Secondary not configured Primary*: 198.51.100.20 Secondary: 0.0.0.0 Token Expires: Mar 28, 2018 Last status update 1 second(s) ago (*) this Management Center
    Please record authentication token for setup with primary and press Enter.
    Because the secondary failover partner has not been configured, the failover icon displays with an exclamation mark:
    This icon also displays if failover has been configured and the secondary is unresponsive.
Step 2—Configure the Secondary Appliance
Before beginning this procedure, complete all tasks required for the secondary appliance to service requests (set up authentication, etc.).
  1. Use an SSH client to log into the CLI of the Management Center appliance that is to be the secondary failover partner.
  2. Enter Enable mode:
    #
    enable
  3. Enter Configuration mode:
    #
    configure
  4. Confirm that failover has not already been configured on the appliance:
    (config)#
    failover view
    Failover:
    Status: Disabled
  5. Enter the following command to begin configuring the secondary failover partner:
    During this process, the services on both the primary and secondary appliances are unavailable.
    (config)#
    failover make-secondary
  6. Enter the secondary failover partner's primary IP address.
    If you intend to use automatic failover, you must enter the virtual IP address of the primary partner as shown below. If you are using manual failover, use a free IP address.
    Manual failover:
    Value for 'primary-ip' (<IP address>):
    198.51.100.20
    Automatic failover:
    Value for 'primary-ip' (<IP address>):
    primary_partner_virtual_ip_address
  7. Enter the token generated by the primary appliance during failover configuration. See Step 1—Configure the Primary Appliance
    Value for 'token' (): ****************
  8. Specify the type of failover mode, automatic or manual.
    Value for 'failover mode' (automatic, manual)
    If you specify
    automatic
    , you must now choose a timeout value:
    Please enter the minutes to wait if primary goes down before automatically switching to secondary. [1 - 120] minutes?
    Warning: Initial failover data transfer may take a long time to complete. To complete the failover setup, allow for transfer to finish and do not disable failover on 198.51.100.20 (primary) or 198.51.100.16 (secondary) during this operation. Services on 198.51.100.20 (primary) will not be available while initial failover setup is performed.
    Are you sure you want to continue?
    y
    Shelving operational data on secondary...done.
    Stopping services on secondary...done.
    Stopping services on primary...done.
    Retrieving snapshot of primary's data...
    The password is not saved and is not reused for further replication process.
  9. Verify that failover has been successfully configured:
    (config)#
    failover view
    Status: Healthy (1 second replication delay) Primary: 192.0.2.56 Secondary*: 192.0.2.34 Replicating: ACL Configuration: false Authentication Configuration: false Diagnostics Configuration: true Last status update 11 second(s) ago (*) this Management Center
    If failover has been successfully configured, the failover icon displays in the web UI banner:
    You can also mouse over the failover icon to review the failover status.
  10. Optional—Use the CLI to configure other replication settings.
    #
    failover replicate ?
    See failover for more information.

Manual Failover - Switch to Secondary When the Primary is Unresponsive

If the primary failover partner is unresponsive and you have not configured automatic failover, you must do the following:

On the Secondary Failover Partner:

  1. Log into the secondary failover partner using the CLI admin account of the device. See Add Local Users for more information.
  2. Enter Configuration mode and make the secondary failover partner active. Do this by entering the command:
  3. (config)#
    failover make-primary
  4. Reactivate statistics monitoring.
At this point, the secondary is active and is now the primary failover partner. For details, see Step 1—Make Secondary Partner Active and Step 2—Reactivate Statistics Monitoring

On the Original Primary Device:

  1. Fix the problem on the original primary failover partner.
  2. On the original primary failover partner, enter Configuration mode and make it (the device that was unresponsive) the new secondary failover partner:
    (config)#
    failover make-secondary
Failover is now successfully reconfigured.
Step 1—Make Secondary Partner Active
Issue the
failover make-primary
command to make the secondary appliance the primary failover partner. If the original primary device later becomes responsive, you can make it the secondary failover partner, thus preserving the failover capability.
(config)#
failover make-primary
System is configured as secondary, promoting state to primary will break replication.
Are you sure you want to promote state to primary? [y/N]
Restoring operational data...done.
Failover:
Status: ERROR: Secondary not configured
Primary*: 198.51.100.24
Secondary: not configured
Last status update 2 second(s) ago
(*) this Management Center
Step 2—Reactivate Statistics Monitoring
After making the secondary failover partner active, you must reactive the statistics monitoring job. This job instructs devices that have PDM Export (statistics monitoring) enabled to send updates to the new primary device.
  1. Select
    Jobs > Scheduled Jobs
    .
  2. Click
    New Job
    . The system displays the New Job: Basic Info dialog.
  3. In the Basic Info dialog, enter a name for your job.
  4. Enter a description of the job. Good descriptions help to differentiate jobs when they have similar names.
  5. Click
    Next
    .
  6. In the Operation dialog, select
    Reactivate Statistics Monitoring
    .
  7. Click
    Next
    .
    The system displays the
    Targets
    dialog. Management Center automatically finds all applicable targets.
  8. Click
    Next
    .
    The system displays the
    Schedule
    dialog. Optionally, enter a schedule.
  9. Click
    Finish
    .

Upgrade the Failover Pair

Upgrading is a complex procedure. Please review the document Upgrade
Management Center
before starting this procedure.
Step 1 - Back Up the Primary Partner
Before upgrading your failover pair, back up the Primary partner's configuration and export it off-box to a secure location. See Back Up the Management Center Configuration. You do not need to back up the Secondary partner.
Step 2 - Disable Failover on the Primary and Secondary Partners
Complete the following procedure on both the Primary and Secondary failover partners
before
upgrading.
  1. Log into theCLI and enter configuration mode.
  2. Enter the following command:
    (config)#
    failover
  3. Enter the following command to disable the Primary partner:
    (config-failover)#
    disable
    The system terminates the CLI session when you run the
    disable
    command.
  4. Log back into the CLI and go step 5.
  5. Enter the following command to verify that failover has been disabled.
    (config-failover)#
    view
Step 3 - Download the System Image on Both Primary and Secondary Partners
Download the desired image on both the Primary and Secondary partners:
  • Transfer the image directly to
    Management Center
    . Select
    Configuration > Files
    and transfer the image using the Transfer File button.
  • Download the image to a local drive, select
    Configuration > Files
    , and upload the image to
    Management Center
    .
Alternatively, you can store the image file on a web server that the
Management Center
appliance can access. The add image process works with any HTTP server, and HTTPS servers configured with trusted certificates. If your HTTPS server does not have a trusted certificate, place the file on an internal HTTP server.
If you require HTTP service, enable it using the following command:
(config)#
security http enable
For security reasons, you should immediately disable the HTTP service after retrieving the system image.
Step 4 - Upgrade Both Primary and Secondary Partners
Follow the procedures in Upgrade
Management Center
to upgrade the Primary and Secondary partners.
Step 5 - Re-enable Failover on Both Primary and Secondary Partners
Follow the procedures in Configure Management Center Failover to re-enable failover on the Primary and Secondary partners.
 

Configure SNMP Alert or SMTP Trap for Failover Alerts

You can configure an SNMP alert and/or SMTP trap to notify you if there is an error in the failover state. When configured, the health check runs every 60 seconds to determine the health of the failover configuration. Possible error states include:
  • Secondary not configured
  • Host not configured as failover partner
  • Partner IP address is not active
If the health check encounters a failover error, it checks two more times before it confirms the error. If the health check confirms the error, a trap is sent and the health check will resume. When it detects that the failover pair is healthy again, the health check sends another trap indicating that the problem has been resolved.
The failover health check can be configured only from the
Management Center
CLI. You cannot configure this option using the user interface.
Configure Failover Health SNMP Trap
To receive an SNMP trap when the health check detects a failover error, you must configure the following from the
Management Center
CLI: SNMP community name, vacm view, vacm, group, vacm access, notify target name, and notify target IP address. Then, you must enable the SNMP agent.
Example SNMP Trap Configuration
(config)#
snmp community public
(config-community-public)#
exit
(config-snmp)#
vacm view bc subtree 1.3 included
(config-snmp)#
vacm group public member public sec-model v2c
(config-snmp)#
vacm group public access v2c no-auth-no-priv notify-view bc read-view bc write-view bc
(config-snmp)#
notify target1 type trap tag target1
(config-snmp)#
target target1 ip 192.0.2.14 tag target1 udp-port 162 v2c sec-name public
(config-snmp)#
agent enable
The SNMP failover health trap uses the BLUECOAT-SG-HEALTHMONITOR-MIB, which is included in the BLUECOAT-MIB. You can download these MIBs on the Download site.
Configure Failover Health SMTP Trap
To receive an SMTP trap when the health check detects a failover error, you must configure the following from the
Management Center
CLI:
(config)#
smtp
(config-smtp)#
destination-address add
address
(config-smtp)#
from-address
address
(config-smtp)#
gateway
gateway

Example SNMP Trap Configuration

(config)#
snmp community public
(config-community-public)#
exit
(config-snmp)#
vacm view bc subtree 1.3 included
(config-snmp)#
vacm group public member public sec-model v2c
(config-snmp)#
vacm group public access v2c no-auth-no-priv notify-view bc read-view bc write-view bc
(config-snmp)#
notify target1 type trap tag target1
(config-snmp)#
target target1 ip 192.0.2.14 tag target1 udp-port 162 v2c sec-name public
(config-snmp)#
agent enable
The SNMP failover health trap uses the BLUECOAT-SG-HEALTHMONITOR-MIB, which is included in the BLUECOAT-MIB. You can download these MIBs on the Download site.

View Failover Health Check Logs

View failover health check logs in the following location:
/var/log/user_syslog

Disable Failover

Use the
failover disable
command to disable failover.
(config)#
failover disable
Failover:
Status: Healthy (0 second replication delay)
Primary: 198.51.100.20
Secondary*: 198.51.100.24
Last status update 2 second(s) ago
(*) this Management Center
Are you sure you want to disable failover? [y/N]
Restoring operational data...done.
Failover:
Status: Disabled