Configure to Run as a Cluster

You can have 
Layer7 Live API Creator
 scale and have it provide high availability of services by configuring  to run as a cluster of nodes.
lac53
You can have 
Layer7 Live API Creator
 scale and have it provide high availability of services by configuring 
Layer7 Live API Creator
 to run as a cluster of nodes. Run 
Layer7 Live API Creator
 as a cluster so that your APIs can handle request load and perform optimally. 
Layer7 Live API Creator
 runs as a stateless service: a node processes each API request separately, without remembering the state from previous API requests.
Running 
Layer7 Live API Creator
 as a cluster requires that the following systems work together:
  • Multiple 
    Layer7 Live API Creator
     instances, running in Java containers (for example, Jetty or Apache Tomcat) or as Docker containers. For redundancy, run these 
    Layer7 Live API Creator
     instances on physically separate hardware so that if a machine becomes unavailable, the others run.
  • A load balancer, such as 
    Layer7 API Gateway
     or Amazon Elastic Load Balancing. The load balancer distributes incoming request traffic to the nodes in the cluster.
    For more information about how to use the Gateway as a load balancer, see the information about how to specify how the Gateway should retrieve IP addresses on the [Connection] tab for the 
    Route via HTTP(S)
     assertion in the 
    Layer7 API Gateway
     documentation
    .
Configure 
Layer7 Live API Creator
 to run as a cluster using the following process:
Verify the Prerequisites
Before configuring 
Layer7 Live API Creator
 to run as a cluster, verify the following prerequisites:
  • You have verified that you have the IP address or server DNS name of all nodes.
  • If you are licensed to use 
    Layer7 Live API Creator
     under the standard license that the CA Support site issues, you have completed the following:
    • You have the license for the newer version of 
      Layer7 Live API Creator
      .
    • If you have consented to 
      Layer7 Live API Creator
       collecting and sending usage data in other environments where you have installed 
      Layer7 Live API Creator
       and you want to apply the same usage data configuration to the cluster, you have updated the following attributes that are in the file that 
      Layer7 Live API Creator
       saves your usage data settings (the 
      system/telemetry.json
       file). This file is part of your admin repository:
      • pla_enabled:
         You have set this attribute to 
        false
         (you are licensed to use 
        Layer7 Live API Creator
         under a standard license).
      • send_enabled: 
        You have set this attribute to 
        true
         (you have consented to 
        Layer7 Live API Creator
         sending usage data). By default, 
        Layer7 Live API Creator
         does not collect and send usage data (the value is 
        false
        ).
      Updating this file is part of the DevOps workflow. For more information about this step in the workflow, see Team Development.
  • If you are licensed to use 
    Layer7 Live API Creator
     under an Enterprise Software Portfolio License Agreement (PLA), you have completed the following:
    • You have verified that the admin repository includes the 
      system/telemetry.json
       file.
    • You have updated the following attributes that are in the file that 
      Layer7 Live API Creator
       saves your usage data settings (the 
      system/telemetry.json
       file). This file is part of your admin repository:
      •  
        pla_enabled:
         You have set this attribute to 
        true
         (you are licensed to use 
        Layer7 Live API Creator
         under a PLA).
      •  
        send_enabled:
         You have set this attribute to 
        true
         (you have consented to 
        Layer7 Live API Creator
         collecting and sending usage data). By default, 
        Layer7 Live API Creator
         does not collect and send usage data (the value is 
        false
        ).
      •  
        domain_name:
         You have set this attribute to the last part of your company's email address (the domain).
      •  
        site_id:
         You have set this attribute to the enterprise site ID that CA Support assigns to your company. The ID is visible when you log in to the CA Support site.
      •  
        chargeback_id: 
        (optional) You have set this attribute to the department or cost center that you use for internal tracking, such as the ID for billing back to groups.
    For more information about licensing, see Licensing.
Create a Database for your Authentication Tokens
In cluster configurations, 
Layer7 Live API Creator
 authenticates API users across the nodes in the cluster by sharing the dynamically-created authentication tokens that it stores in the database that you create for the authentication tokens.
Your 
Layer7 Live API Creator
 cluster configuration requires that you create a database for your authentication tokens if 
any
 of the following cases are true:
  • You have specified an authentication provider that uses the 
    Default Auth Provider
     authentication  method (for example, the 
    built-in authentication
     authentication provider) or the 
    JavaScript Auth Provider
     authentication method as the authentication provider for your API.
  • You are not securing your published APIs using API Gateway.
For more information about how to create this database, see Create a Database for your Authentication Tokens.
Configure External Logging
You can configure external logging for API logs, for listener execution logs, for timer execution logs, and more. You can use these logs to troubleshoot issues, such as API error and error frequency, as well as listener and timer issues.
For more information about how to configure external logging, see External Logging.
Add Each Node to your Load Balancer Configuration
In your Java container, this can be an auto scale setting up to a maximum number of nodes in the cluster configuration.
For more information about how to configure the load balancer for a cluster of application servers, consult your load balancer documentation. 
You likely need the IP address or server DNS name of all the nodes in the cluster to add them to your load balancer configuration.
Deploy your APIs to a Cluster
Deploy your APIs to a cluster using 
one
 of the following approaches:
Deploy from a URL or Git
Prerequisite: 
You have 
Layer7 Live API Creator
 installed on each node.
Add the following options when you start API Server:
Option
Description
LAC_REPOSITORY_CONFIGURATION_URL
 
This option sets the location (URL or Git) from which 
Layer7 Live API Creator
 pulls your admin repository.
LAC_REPOSITORY_DISABLE
When you set this option to 
false
, API Server does not write to disk the changes that you make to the admin repository.
Each node runs in its own admin repository. To prevent inconsistent behavior across the cluster while running in a cluster, do not create new API definitions or modify existing ones.
Values:
 
true
 or 
false
 
Default:
 
false
 
Best Practice:
 When running 
Layer7 Live API Creator
 in a production environment, set this option to 
false
.
LAC_LOGGING_CONFIG_FILE
This option sets the absolute path of the logging configuration file.
Required: 
Yes if you have configured external logging.
LAC_CLUSTER_SYNC_STRATEGY
Required: 
Yes if you plan to have 
Layer7 Live API Creator
 execute timers or listeners on a node in the cluster instead of on all nodes in the cluster.
LAC_USE_IN_MEMORY_LOGGING
When you set this option to 
false
, this option turns off in-memory logging.
Values:
 
true
 or 
false
Default:
 
true
 
Best Practice:
 Turn off in-memory logging at the API Server level, especially in test and production environments, and then configure external logging.
For more information about this installation best practice and the other installation best practices, see Installation Best Practices.
Example:
 
<...Tomcat startup...> -DLAC_REPOSITORY_CONFIGURATION_URL= https://s3-us-west-1.amazonaws.com/mybucket/myRepository.zip -DLAC_REPOSITORY_DISABLE=true -DLAC_LOGGING_CONFIG_FILE=/<path_to_your_logging_configuration_file>/<configuration_file_name> -DLAC_CLUSTER_SYNC_STRATEGY=in_memory_with_locking -DLAC_USE_IN_MEMORY_LOGGING=false
For more information about these options, see API Server Startup Options.
You have deployed 
Layer7 Live API Creator
 from a URL or Git.
Deploy from a WAR File that is Bundled with the Admin Repository
  1. Bundle your admin repository ZIP file into the 
    /development/CALiveAPICreator.war
     file.
    For an example of how to do this, see Install on Apache Tomcat.
  2. Install 
    Layer7 Live API Creator
     (the WAR file) on each node based on the Java container type that you are running.
    For more information, see Installing and Upgrading.
  3. Add the following options when you start API Server:
    Option
    Description
    LAC_REPOSITORY_DISABLE
    This option prevents API Server from writing to disk the changes that you make to the admin repository.
    Each node runs in its own admin repository. To prevent inconsistent behavior across the cluster while running in a cluster, do not create new API definitions or modify existing ones.
    Recommended:
     Yes
    LAC_DEFAULT_LICENSE_FILE
    This option sets the location of your license file.
     
    Required:
     Yes if you are licensed to use 
    Layer7 Live API Creator
     under a standard license.
    CA_ACCEPT_LICENSE
    This option bypasses the API Creator screen that asks the initial API developer (a TeamSpace user) to log in to accept the user license, which pre-accepts the end-user license agreement (EULA).
    Required: 
    Yes if you are licensed to use 
    Layer7 Live API Creator
     under a standard license.
    LAC_LOGGING_CONFIG_FILE
    This option sets the absolute path of the logging configuration file.
    Required: 
    Yes if you have configured external logging.
    LAC_REPOSITORY_ROOT
    This option sets the location of your admin repository.
     
    Required:
     No
    LAC_CLUSTER_SYNC_STRATEGY
    Required: 
    Yes if you plan to have 
    Layer7 Live API Creator
     execute timers or listeners on a node in the cluster instead of on all nodes in the cluster.
    hazelcast.config
    This option assigns the path of the Hazelcast configuration XML file when determining how nodes discover each other in a cluster.
    Required:
     No
 
Example:
 
<...Tomcat startup...> -DLAC_REPOSITORY_DISABLE=true -DLAC_DEFAULT_LICENSE_FILE=/Users/jdoe/LACLicense.txt -DCA_ACCEPT_LICENSE=ENU -DLAC_LOGGING_CONFIG_FILE=/<path_to_your_logging_configuration_file>/<configuration_file_name> -DLAC_REPOSITORY_ROOT=${HOME}/CALiveAPICreator.repository -DLAC_CLUSTER_SYNC_STRATEGY=in_memory_with_locking -Dhazelcast.config=/foo/bar/hazelcast.xml
For more information about these options, see API Server Startup Options.
You have deployed 
Layer7 Live API Creator
 from a WAR file that is bundled with the admin repository.
Advanced Configuration
The following topics provide advanced configuration details.
Configure Hazelcast for Cluster Synchronization Strategy
If you plan to have 
Layer7 Live API Creator
 execute timers or listeners for messaging support that rely on cluster synchronization, you must configure Hazelcast for cluster synchronization strategy. Hazelcast is an in-memory data grid for clustering and highly scalable data distribution. 
Layer7 Live API Creator
 uses Hazelcast for coordinating the synchronization among the cluster of nodes. Hazelcast shares data around the cluster for flexibility and performance. Each node in the cluster is functionally the same. The nodes discover each other within the Hazelcast cluster. Hazelcast clusters do not have a master node and, therefore no single point of failure in a Hazelcast cluster.   
You configure Hazelcast for cluster synchronization strategy by adding the 
LAC_CLUSTER_SYNC_STRATEGY
 option when you start API Server, with 
one
 of the following property values:
Cluster Synchronization Strategy
Property Value
Description
Nodes assign themselves identification numbers that are based on the order in which the Hazelcast cluster initializes the nodes. The nodes decide using these ID numbers.
in_memory
Because the nodes decide without the use of shared objects, operations that use cluster synchronization perform better with this strategy.
Cluster disruptions, for example, a node in the cluster dropping out unexpectedly, can cause 
Layer7 Live API Creator
 to process messages in listeners or timer code execution more than once or not at all during the disruption, which typically last no longer than a few seconds.
Layer7 Live API Creator
 uses a shared map to hold a mutex that the active nodes refer to before making decisions.
in_memory_with_locking
This strategy performs with high reliability. Because some nodes wait on locked objects, this strategy affects the performance of the underlying operations that rely on cluster synchronization, such as listeners or timers.
Best Practice:
 If reliability is more important than performance or if you are configuring 
Layer7 Live API Creator
 to run in a cluster that is unstable or that changes in size frequently, use this strategy.
Build your Hazelcast Configuration XML File
Hazelcast nodes automatically join to form a cluster. This automatic joining takes place with various node discovery mechanisms that the cluster members use to find each other.
For more information about the Hazelcast node discovery mechanisms, including how to configure them, see the Hazelcast documentation.
Build your Hazelcast configuration XML file based on the environment and supported discovery mechanisms on which you want to run 
Layer7 Live API Creator
:
Run in an Amazon Web Services Elastic Beanstalk (AWS EBS) Environment
You can use the  file as an example for installing in an AWS EBS environment.
  1. Open the 
    hazelcast.xml
    , and then open the ports for inbound and outbound connection for the nodes by editing the port numbers in the following section of the file:
    <port auto-increment="true" port-count="100">5701</port>
    <outbound-ports>
    <ports>5900-6750</ports>
    </outbound-ports>
  2. In AWS, configure the AWS EBS security group to allow communication between nodes through the configured nodes.
  3. Complete 
    one
     of the following:
    • In the 
      hazelcast.xml
       file, add the AWS access key, AWS secret key, and the names of the nodes in the cluster in the following section of the file, and then save and close the file:
      <discovery-strategies>
      <discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
      <properties>
      <property name="access-key">###AWSACCESSKEY</property>
      <property name="secret-key">###AWSSECRETKEY</property>
      <property name="tag-key">Name</property>
      <property name="tag-value">##NAME OF THE ELASTIC BEANSTALK ENVIRONMENT</property>
      </properties>
      </discovery-strategy>
      </discovery-strategies>
    • Do the following:
      1. In the 
        hazelcast.xml
         file, provide an Identity and Access Management (IAM) role and the names of the nodes in the cluster in the following section of the file, and then save and close the file:
        <discovery-strategies>
        <discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
        <properties>
        <property name="iam-role">HazelCast_Role</property>
        <property name="tag-key">Name</property>
        <property name="tag-value">##NAME OF THE ELASTIC BEANSTALK ENVIRONMENT</property>
        </properties>
        </discovery-strategy>
        </discovery-strategies>
      2. In AWS, define the following permission policy, at minimum, to your IAM user:
        "ec2:DescribeInstances"
        {
        "Version": "XXXXXXXX",
        "Statement": [
        {
        "Sid": "XXXXXXXX",
        "Action": [
        "ec2:DescribeInstances"
        ],
        "Effect": "Allow",
        "Resource": "*"
        }
        ]
        }
Run by connecting directly by way of Transmission Control Protocol (TCP)
You can use the  file as an example for connecting directly by way of TCP. Open the 
hazelcast_tcp_example.xml
 file, enter the IP and port for all nodes in the cluster in the following section of the file, and then save and close the file:
<tcp-ip enabled="true">
<member-list>
<member>#NODE-IP-1#:#NODE-PORT#</member>
<member>#NODE-IP-2#:#NODE-PORT#</member>
</member-list>
</tcp-ip>
Run in an environment other than AWS or other than connecting directly by way of TCP
Run in an environment other than AWS or other than connecting directly by way of TCP. You can use the  file as a reference. This file shows the various elements of a Hazelcast configuration.
For more information about how to set up Hazelcast clusters, see the Hazelcast documentation.
Configure Hazelcast External Logging
You can externalize Hazelcast logs by configuring Hazelcast external logging. Use these logs to troubleshoot and verify the Hazelcast configuration. Configure this logging while configuring external logging by including the 
lacclustersynclogger
 logger in the logging configuration file that you create.
For more information about how to configure Hazelcast external logging, see External Logging.
Determine How the Nodes Discover Each Other in a Cluster
Determine how the nodes discover each other in a cluster by completing 
one
 of the following steps:
  • On all the nodes in the cluster, add your Hazelcast configuration XML file to the 
    Layer7 Live API Creator
     
    classpath
    .
  • Copy your Hazelcast configuration XML file to a shared directory that the cluster can access, and then, when you start API Server, assign the path of this file by adding the 
    hazelcast.config
     option.
    For more information about this option, see the "Deploy from a WAR file bundled with the admin repository" section.
Next Steps
After you have configured 
Layer7 Live API Creator
 to run as a cluster, you can test the configuration by calling the endpoints through your load balancer.