Configure Live API Creator to Run as a Cluster

You can have 
CA Live API Creator
 scale and have it provide high availability of services by configuring 
CA Live API Creator
 to run as a cluster of nodes.
lac42
You can have 
CA Live API Creator
 scale and have it provide high availability of services by configuring 
CA Live API Creator
 to run as a cluster of nodes. 
CA Live API Creator
 runs as a stateless service: a node processes each API request separately, without remembering the state from previous API requests.
Running 
CA Live API Creator
 as a cluster requires that the following systems work together:
  • Multiple 
    CA Live API Creator
     instances, running in Java containers (for example, Jetty or Apache Tomcat) or as Docker containers. For redundancy, run these 
    CA Live API Creator
     instances on physically separate hardware so that if a machine becomes unavailable, the others run. 
  • A load balancer, such as
    CA API Gateway
    or Amazon Elastic Load Balancing. The load balancer distributes incoming request traffic to the nodes in the cluster.
    For more information about how to use
    CA API Gateway
    as a load balancer, see the information about how to specify how the Gateway should retrieve IP addresses on the [Connection] tab for the 
    Route via HTTP(S)
     assertion in the
    CA API Gateway
    documentation
    .
Configure 
CA Live API Creator
 to run as a cluster using the following process:
Verify the Prerequisite
Before you configure 
CA Live API Creator
 to run as a cluster, ensure that you have the IP address or server DNS name of all nodes.
Create a Database for your Authentication Tokens
In cluster configurations, 
CA Live API Creator
 authenticates API users across the nodes in the cluster by sharing the authentication tokens that it stores in the database that you create for the authentication tokens.
Your 
CA Live API Creator
 cluster configuration requires that you create a database for your authentication tokens if 
any
 of the following cases are true:
  • You have specified an authentication provider that uses the 
    Default Auth Provider
     authentication method (for example, the 
    built-in authentication
     authentication provider) or the 
    JavaScript Auth Provider
     authentication method as the authentication provider for your API.
  • You are not securing your published APIs using API Gateway.
For more information about how to create this database, see Create a Database for your Authentication Tokens.
Configure External Logging
You can configure external logging for API logs, for listener execution logs, for timer execution logs, and more. You can use these logs to troubleshoot issues, such as API error and error frequency, as well as listener and timer issues.
For more information about how to configure external logging, see External Logging.
Add Each Node to your Load Balancer Configuration
In your Java container, this can be an auto scale setting up to a maximum number of nodes in the cluster configuration.
For more information about how to configure the load balancer for a cluster of application servers, consult your load balancer documentation. 
 You likely need the IP address or server DNS name of all the nodes in the cluster.
Deploy your APIs to a Cluster
Deploy your APIs to a cluster using 
one
 of the following approaches:
Deploy from a URL or Git
 
Prerequisite: 
You have 
CA Live API Creator
 installed on each node.
This is the recommended deployment approach.
Add the following options when you start API Server:
Option
Description
 
LAC_REPOSITORY_CONFIGURATION_URL
 
This option sets the location (URL or Git) from which 
CA Live API Creator
 pulls your admin repository.
 
LAC_REPOSITORY_DISABLE
 
This option prevents API Server from writing to disk the changes that you make to the admin repository.
 Each node runs in its own admin repository. To prevent inconsistent behavior across the cluster while running in a cluster, do not create new API definitions or modify existing ones.
 
Recommended:
 Yes
 
LAC_LOGGING_CONFIG_FILE
 
This option sets the absolute path of the logging configuration file.
 
Required: 
If you have configured external logging, then yes.
 
LAC_CLUSTER_SYNC_STRATEGY
 
 
Required: 
If you plan to have 
CA Live API Creator
 execute timers or listeners on a node in the cluster instead of on all nodes in the cluster, then yes.
 
Example:
 
<...Tomcat startup...> -DLAC_REPOSITORY_CONFIGURATION_URL= https://s3-us-west-1.amazonaws.com/mybucket/myRepository.zip -DLAC_REPOSITORY_DISABLE=true -DLAC_LOGGING_CONFIG_FILE=/<path_to_your_logging_configuration_file>/<configuration_file_name> -DLAC_CLUSTER_SYNC_STRATEGY=in_memory_with_locking
For more information about these options, see API Server Startup Options.
You have deployed 
CA Live API Creator
 from a URL or Git.
Deploy from a WAR File that is Bundled with the Admin Repository
 
Prerequisite: 
You have the license for the newer version of 
CA Live API Creator
.
 
Follow these steps:
 
  1. Bundle your admin repository ZIP file into the 
    /development/CALiveAPICreator.war
     file.
    For an example of how to do this, see Install on Apache Tomcat.
  2. Install 
    CA Live API Creator
     (the WAR file) on each node based on the Java container you are running.
    For more information, see Installing and Upgrading.
  3. Add the following options when you start API Server:
    Option
    Description
    LAC_REPOSITORY_DISABLE
    This option prevents API Server from writing to disk the changes that you make to the admin repository.
     Each node runs in its own admin repository. To prevent inconsistent behavior across the cluster while running in a cluster, do not create new API definitions or modify existing ones.
     
    Recommended:
     Yes
    LAC_DEFAULT_LICENSE_FILE
    This option sets the location of your license file.
     
    Required:
     Yes
    ca_accept_license
    This option bypasses the API Creator screen that asks the initial TeamSpace user to log in to accept the user license, which pre-accepts the End User License Agreement (EULA).
     
    Required:
     Yes
    LAC_LOGGING_CONFIG_FILE
    This option sets the absolute path of the logging configuration file.
     
    Required: 
    If you have configured external logging, then yes.
    LAC_REPOSITORY_ROOT
    This option sets the location of your admin repository.
     
    Required:
     No
     
    LAC_CLUSTER_SYNC_STRATEGY
     
    This option configures Hazelcast for cluster synchronization strategy.
    Required: 
    If you plan to have 
    CA Live API Creator
     execute timers or listeners on a node in the cluster instead of on all nodes in the cluster, then yes.
     
    hazelcast.config
     
    This option assigns the path of the Hazelcast configuration XML file when determining how nodes discover each other in a cluster.
     
    Required:
     No
 
Example:
 
<...Tomcat startup...> -DLAC_REPOSITORY_DISABLE=true -DLAC_DEFAULT_LICENSE_FILE=/Users/jdoe/LACLicense.txt -Dca_accept_license=ENU -DLAC_LOGGING_CONFIG_FILE=/<path_to_your_logging_configuration_file>/<configuration_file_name> -DLAC_REPOSITORY_ROOT=${HOME}/CALiveAPICreator.repository -DLAC_CLUSTER_SYNC_STRATEGY=in_memory_with_locking -Dhazelcast.config=/foo/bar/hazelcast.xml
For more information about these options, see API Server Startup Options.
You have deployed 
CA Live API Creator
 from a WAR file that is bundled with the admin repository.
Advanced Configuration
The following topics provide advanced configuration details.
Configure Hazelcast for Cluster Synchronization Strategy
If you plan to have 
CA Live API Creator
 execute timers or listeners for messaging support that rely on cluster synchronization, you must configure Hazelcast for cluster synchronization strategy. Hazelcast is an in-memory data grid for clustering and highly scalable data distribution. 
CA Live API Creator
 uses Hazelcast for coordinating the synchronization among the cluster of nodes. Hazelcast shares data around the cluster for flexibility and performance. Each node in the cluster is functionally the same. The nodes discover each other within the Hazelcast cluster. Hazelcast clusters do not have a master node and, therefore no single point of failure in a Hazelcast cluster.   
You configure Hazelcast for cluster synchronization strategy by adding the 
LAC_CLUSTER_SYNC_STRATEGY
 option when you start API Server, with 
one
 of the following property values:
Cluster Synchronization Strategy
Property Value
Description
Nodes assign themselves identification numbers that are based on the order in which the Hazelcast cluster initializes the nodes. The nodes decide using these ID numbers.
in_memory
Because the nodes decide without the use of shared objects, operations that use cluster synchronization perform better with this strategy.
 Cluster disruptions, for example, a node in the cluster dropping out unexpectedly, can cause 
CA Live API Creator
 to process messages in listeners or timer code execution more than once or not at all during the disruption, which typically last no longer than a few seconds.
CA Live API Creator
 uses a shared map to hold a mutex that the active nodes refer to before making decisions.
in_memory_with_locking
This strategy performs with high reliability. Because some nodes wait on locked objects, this strategy affects the performance of the underlying operations that rely on cluster synchronization, such as listeners or timers.
Best Practice:
 If reliability is more important than performance or if you are configuring 
CA Live API Creator
 to run in a cluster that is unstable or that changes in size frequently, use this strategy.
Build your Hazelcast Configuration XML File
Hazelcast nodes automatically join to form a cluster. This automatic joining takes place with various node discovery mechanisms that the cluster members use to find each other.
For more information about the Hazelcast node discovery mechanisms, including how to configure them, see the Hazelcast documentation.
Build your Hazelcast configuration XML file based on the environment and supported discovery mechanisms on which you want to run 
CA Live API Creator
:
  •  
    Run in an Amazon Web Services Elastic Beanstalk (AWS EBS) Environment
    You can use the  file as an example for installing in an AWS EBS environment.
    • Open the 
      hazelcast.xml
       file, and then open the ports for inbound and outbound connection for the nodes by editing the port numbers in the following section of the file:
      <port auto-increment="true" port-count="100">5701</port>
      <outbound-ports>
      <ports>5900-6750</ports>
      </outbound-ports>
    • In AWS, configure the AWS EBS security group to allow communication between nodes through the configured nodes.
    • Complete 
      one
       of the following:
      • In the 
        hazelcast.xml
         file, add the AWS access key, AWS secret key, and the names of the nodes in the cluster in the following section of the file, and then save and close the file:
        <discovery-strategies>
        <discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
        <properties>
        <property name="access-key">###AWSACCESSKEY</property>
        <property name="secret-key">###AWSSECRETKEY</property>
        <property name="tag-key">Name</property>
        <property name="tag-value">##NAME OF THE ELASTIC BEANSTALK ENVIRONMENT</property>
        </properties>
        </discovery-strategy>
        </discovery-strategies>
      • Do the following:
        1. In the 
          hazelcast.xml
           file, provide an Identity and Access Management (IAM) role and the names of the nodes in the cluster in the following section of the file, and then save and close the file:
          <discovery-strategies>
          <discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
          <properties>
          <property name="iam-role">HazelCast_Role</property>
          <property name="tag-key">Name</property>
          <property name="tag-value">##NAME OF THE ELASTIC BEANSTALK ENVIRONMENT</property>
          </properties>
          </discovery-strategy>
          </discovery-strategies>
        2. In AWS, define the following permission policy, at minimum, to your IAM user:
          "ec2:DescribeInstances"
          {
          "Version": "XXXXXXXX",
          "Statement": [
          {
          "Sid": "XXXXXXXX",
          "Action": [
          "ec2:DescribeInstances"
          ],
          "Effect": "Allow",
          "Resource": "*"
          }
          ]
          }
  •  
    Run by connecting directly by way of Transmission Control Protocol (TCP)
    You can use the  file as an example for connecting directly by way of TCP. Open the
    hazelcast_tcp_example.xml
    file, and then enter the IP and port for all nodes in the cluster in the following section of the file:
    <tcp-ip enabled="true">
    <member-list>
    <member>#NODE-IP-1#:#NODE-PORT#</member>
    <member>#NODE-IP-2#:#NODE-PORT#</member>
    </member-list>
    </tcp-ip>
  •  
    Run in an environment other than AWS or other than connecting directly by way of TCP
    Run in an environment other than AWS or other than connecting directly by way of TCP. You can use the  file as reference. This file shows the various elements of a Hazelcast configuration.
    For more information about how to set up Hazelcast clusters, see the Hazelcast documentation.
Configure Hazelcast External Logging
You can externalize Hazelcast logs by configuring Hazelcast external logging. Use these logs to troubleshoot and verify the Hazelcast configuration. Configure this logging while configuring external logging by including the 
lacclustersynclogger
 logger in the logging configuration file that you create.
For more information about how to configure Hazelcast external logging, see External Logging.
Determine How the Nodes Discover Each Other in a Cluster
Determine how the nodes discover each other in a cluster by completing 
one
 of the following steps:
  • On all the nodes in the cluster, add your Hazelcast configuration XML file to the 
    CA Live API Creator
     
    classpath
    .
  • Copy your Hazelcast configuration XML file to a shared directory that the cluster can access, and then, when you start API Server, assign the path of this file by adding the 
    hazelcast.config
     option.
    For more information about this option, see the "Deploy from a WAR file bundled with the admin repository" section.
Next Steps
After you have configured 
CA Live API Creator
 to run as a cluster, you can test the configuration by calling the endpoints through your load balancer.