High Availability

This topic provides an overview of high availability cluster setup for CA API Portal. 
apip42
This topic provides an overview of high availability cluster setup for CA API Portal. 
Establishing and maintaining a High Availability (HA) APIM platform infrastructure is essential to providing uninterrupted service in production environments. The same HA architecture that provides redundancy in the case of failover can also be used to scale services to handle additional load.
Topics include:
2
2
Prerequisites:
  • A MySQL database cluster.
  • A user with permissions to create databases
  • An external load balancer
Deploy Multiple Swarms
You can deploy multiple portal swarms. However, they must have the same configuration, and the same certificates and keys.
To deploy an additional swarm:
  1. First, deploy CA API Portal in a single cluster.
  2. Create a tar-ball containing all certificates/keys and portal.conf on the configured portal.
    cd /opt/apim-portal-<VERSION> sudo tar czf config.tar.gz certs conf
  3. Create a VM in another data center.
  4. Download the license file, portal package and config tar-ball to the VM:
    scp apim-portal-<VERSION>.tar.gz portal@your-portal-host:/tmp scp LICENSE.xml portal@your-portal-host:/tmp scp config.tar.gz portal@your-portal-host:/tmp
    For multiple development environments, you need licenses for each portal instance.
  5. Extract the portal package:
    sudo tar zxvf /tmp/apim-portal-<VERSION>.tar.gz -C /opt/
  6. Extract the config tar-ball into the portal package:
    cd /opt/apim-portal-<VERSION>/ sudo tar zxvf /tmp/config.tar.gz
  7. Start the portal:
    sudo ./portal.sh
After all the portal instances have started, configure your load balancer to point to these portal IPs. See Load Balancing Best Practices.
Configure the Load Balancer
Load Balancer Requirements:
  • The host name of the load balancer must be different from the host name of the external tenant.
  • The enrollment of an external tenant gateway must be done by hitting the portal directly. To do this, you can use any of the HA portals.
  • You can use any load balancer that supports HTTP and TCP load balancing.
  • The following ports are required for the portal:
    • 443
    • 9443
  • The load balancer should be configured to terminate SSL on port 443. On port 9443, SSL 
    should not
     be terminated. The traffic on port 9443 should be passed directly to the dispatcher of the portals. Optionally, you can choose to redirect all traffic on port 80 to port 443.
  • The healthcheck endpoint used by the load balancer is
    /portalhealth
External Tenant Port 443
The following headers are important and should be sent to the portals by the load balancer for the external tenant on port 443:
  • Host
  • Origin
  • Referer 
For example, if you have a portal with an external tenant ID of "tenant2" and a portal domain of "portal.ca.com", the headers would be set as:
Host tenant2.portal.domain.com
Origin https://tenant2.portal.ca.com
Referer https://tenant2.portal.ca.com
The location header in the response of the portals should be changed to use the load balancer host name, for example, https://tenant2.portal.ca.com/admin/login should be changed to https://load.balancer.host.ca/admin/login 
Internal Tenant Port 443
It is recommended you route this port directly to the portal host. The Admin user of the external tenant has access to the portal APIs. You can either resolve this address directly to the Portal or use the load balancer. For this to work through the load balancer, you should set the following headers and send the headers to the portals by the load balancer for the external tenant on port 443:
  • Host
  • Origin
  • Referer
Also follow the steps for port 9443.
For example, if you have a portal with an internal tenant ID of "apim" and a portal domain of "portal.ca.com", the headers would be set as:
Host apim.portal.ca.com
Origin https://apim.portal.ca.com
Referer https://apim.portal.ca.com
Leave the internal tenant ID as apim. Set the "'keepalive" timeout on the load balancer to at least 5 seconds. Set the maximum number of requests that can be served through one keep-alive connection should be set to 100.
Port 9443
It is recommended you route this port directly to the portal host. This port is used for analytics, tenant enrollment, API sync, SSO, and portal API access - any of which you can do through the load balancer, as long as you configure these settings explicitly in the load balancer configuration:
  • Set the origin header as shown in the following example:
    Origin "https://apim.portal.ca.com"
  • Communicate to the portal using
    ssl
    and use
    sni
    to set the host name to the expected value. The following examples show appropriate values as host names:
    • enroll.portal.ca.com
    • apim-ssg.portal.ca.com
    • analytics.portal.ca.com
    • sso.portal.ca.com
    • sync.portal.ca.com
    • broker.portal.ca.com
In the previous examples,
portal.ca.com
 is the portal domain. Replace this value with your portal subdomain. 
Health Check Endpoint
The load balancer can access the
/portalhealth 
health check endpoint to discover if a cluster is healthy or not. For example: 
https://<portal-url>/portalhealth
The healthcheck responds on port 443.
An 
HTTP 200 OK
 code is returned if all services on the portal are healthy.
An
HTTP 503
code is returned if any of the services are not available.
The following service status is provided in JSON format: 
  • starting
  • healthy
  • unhealthy
Multiple Cluster Diagram
The following diagram shows one example of a multi-cluster environment. Potential modifications to this architecture include having two separate external databases using replication.  
multiClusterHA
multiClusterHA
Disaster Recovery
To recover the portal and all its data, restore the database using the
util/db-restore.sh
script. It is recommended you copy backups created by the
util/db-backup.sh
script offsite.
For more information, see Back Up and Restore Internal Database.