Connect to an External Hazelcast Datastore

This topic describes how to configure the to use an external in-memory Hazelcast® data store. Such a configuration can improve performance in large data flow scenarios.
gateway94
This topic describes how to configure the
API Gateway
to use an external in-memory Hazelcast® data store. Such a configuration can improve performance in large data flow scenarios.
The sample files provided on this page are not for production use and must be customized to your Gateway configuration. 
Contents:
2
2
What is Hazelcast?
Hazelcast-IMDG-Logo-Orange_Dark_200px.png
Hazelcast is an in-memory data grid service. In a Hazelcast grid, data is distributed evenly between the nodes of a computer cluster, allowing for horizontal scaling of processing and available storage. Backups are distributed among the nodes to protect against failure of any single node. Hazelcast provides central, predictable scaling of applications through in-memory access to frequently-used data and across an elastically scalable data grid. These techniques reduce the query load on databases and improve speed.
You can configure Hazelcast to run in the following configurations:
  • On-premise
  • Cloud (Amazon Web Services, Microsoft Azure, Cloud Foundry, Google Cloud Platform, OpenShift® )
  • Virtual (VMware)
  • Docker containers
Create a Hazelcast Client Configuration File
Creating the Hazelcast client configuration file is required for all Gateway form factors. The file contains the Hazelcast server address plus additional client configuration. 
The Gateway does not support all Hazelcast client configuration parameters. See Supported Hazelcast Client Configuration.
The following is an example of a client-side Hazelcast configuration file
:
<hazelcast-client xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.hazelcast.com/schema/client-config http://www.hazelcast.com/schema/client-config/hazelcast-client-config-3.10.xsd" xmlns="http://www.hazelcast.com/schema/client-config"> <instance-name>gatewayHazelcastClient</instance-name> <network> <cluster-members> <!-- Set your Hazelcast server node addresses here --> <address>hazelcast_host_ip:5701</address> </cluster-members> <connection-attempt-limit>0</connection-attempt-limit> <redo-operation>true</redo-operation> </network> <connection-strategy async-start="false" reconnect-mode="ON" /> </hazelcast-client>
Supported Hazelcast Client Configuration
The 
API Gateway
 relies on the configuration within 
hazelcast-client.xml
. However, the CA API Gateway requires certain parameter values and does not support all configurations of the Hazelcast client.
Configuration elements apply to all Gateway form factors unless noted.
Configuration Parameters
Example
Description
instance-name
<instance-name>gatewayHazelcastClient</instance-name>
The name of the Hazelcast client. The instance-name is required and used by Gateway code.
connection-strategy
<client-connection-strategy async-start="false" reconnect-mode="ON" />
Two attributes in this configuration specify how a Hazelcast client connects to the Hazelcast cluster at startup and after disconnections.
The attributes must be set to the following values:
async-start
:
false
(default value).
Synchronizes the creation of the Hazelcast client with the establishment of a Hazelcast cluster connection. When async-start is set to false, synchronization is required, which ensures that a client/server Hazelcast connection exists before the Gateway starts.
reconnect-mode
:
ON
Reconnection attempts are allowed.
The ASYNC value is not supported and throws exceptions. The OFF value prevents reconnection attempts.
connection-attempt-limit
<connection-attempt-limit>0</connection-attempt-limit>
Defines the number of connection attempts the Hazelcast client makes for any connection scenario. Connection attempts include both startup and reconnection attempts if the client disconnects from the Hazelcast cluster. If a client/server connection is not established with all Hazelcast server nodes within the attempt limit, then the Hazelcast client shuts down.
Recommended value: 0
(zero is used for max number of attempts, equivalent to 2147483647)
Hazelcast documentation recommends tuning connection attempt parameters to define client behavior for Hazelcast cluster failures.
Related configuration for connection-timeout has a default of 5000 ms, which is reasonable. It represents how long a Hazelcast client waits for a server node to respond to its connection request, after which it marks the request as a failed connection attempt.
connection-timeout
<connection-timeout>5000<connection-timeout>
The amount of time a Hazelcast client waits for a server node to respond to its connection request. When the connection-timeout value is exceeded, the request is considered a failed connection attempt. Value is represented in milliseconds.
smart-routing
<smart-routing>true<smart-routing>
The default setting of
true
results in better performance.
This parameter is supported only by Container Gateways on OpenShift. The Hazelcast service and the Gateways must be in the same OpenShift environment.
Smart routing refers to the Hazelcast client being aware of all Hazelcast server nodes, and then sending its requests to the best Hazelcast node (for example, the node owning the data).
redo-operation
<redo-operation>true</redo-operation>
Hazelcast client requests can fail for any number of reasons. Read-only operations are retried by default. If set to
true
, then all operations (read-only and non-read-only) are retried.
Default:
false
near-cache
Not supported.
The
near-cache
setting configures Hazelcast clients to cache data locally on the client.
WARNING:
This configuration is not supported by the Gateway and should not be present. The Gateway will not start up if this setting is present.
Enable External Hazelcast Storage for a Container Gateway
Follow the procedure that corresponds to your deployment of the container Gateway: Docker Engine or OpenShift.
3
3
Docker Engine Environment
The following instructions describe how to enable Hazelcast storage for a Container Gateway running on Docker Engine. 
To enable Hazelcast storage (Docker Engine Environment):
  1. Create the hazelcast-client.xml file locally. See Create a Hazelcast Client Configuration File.
  2. Mount the Hazelcast Client Configuration file to the volumes section inside the docker compose file.
  3. Configure the EXTRA_JAVA_ARGS environment variable to include these entries (line breaks added here for readability):
    EXTRA_JAVA_ARGS:
    "-Dcom.l7tech.server.extension.sharedCounterProvider=externalhazelcast
    -Dcom.l7tech.server.extension.sharedKeyValueStoreProvider=externalhazelcast
    -Dcom.l7tech.server.extension.sharedClusterInfoProvider=externalhazelcast"
The resulting compose file:
version: '2.2' services: api-gateway: image: caapim/gateway ports: - "8080" - "8443" - "9443" volumes: - ./hazelcast/hazelcast-client.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion/hazelcast-client.xml environment: ACCEPT_LICENSE: "false" EXTRA_JAVA_ARGS: "-Dcom.l7tech.server.extension.sharedCounterProvider=externalhazelcast -Dcom.l7tech.server.extension.sharedKeyValueStoreProvider=externalhazelcast -Dcom.l7tech.server.extension.sharedClusterInfoProvider=externalhazelcast"
Copy and paste the example file to create your own 
docker-compose.yml
 file, if you do not have one already. 
Always copy and paste as plain text, to avoid possible reformatting issues. It is recommended that you validate the content in your file before deploying (for example, using a validation site such as yamllint.com).
By setting the 
ACCEPT_LICENSE
 environment variable to “true”, you are indicating you have a valid and existing commercial license for CA API Gateway and you have reviewed and accepted the terms of the CA End User License Agreement (EULA), which governs your use of CA API Gateway.
OpenShift Environment 
The following instructions describe how to enable Hazelcast storage for a Container Gateway running on an OpenShift environment. 
The OpenShift environment has very strict rules for external document mounting. When deploying the container Gateway on an OpenShift environment with external Hazelcast storage, the container image must be repackaged with the hazelcast-client.xml.
To enable Hazelcast storage (OpenShift Environment):
  1. Create the hazelcast-client.xml file locally. See Create a Hazelcast Client Configuration File.
  2. Mount the Hazelcast Client Configuration file to the defined location to enable external Hazelcast storage:
    FROM image:latest
    USER root
    COPY "hazelcast-client.xml" "/opt/SecureSpan/Gateway/node/default/etc/bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion/"
  3. Push the repackaged container Gateway to the cloud repository accessible by OpenShift.
    For example: 
    docker tag <customized gateway image name> <registry-host>/<image-path>
    docker push <registry-host>/<image-path>
Enable External Hazelcast Storage for Other Gateway Form Factors
To configure the all other form factors of the Gateway:
  1. Create the hazelcast-client.xml file locally. See Create a Hazelcast Client Configuration File.
  2. Access the privileged shell.
  3. Navigate to this directory:
    /opt/SecureSpan/Gateway/node/default/etc
  4. Create the following folder structure:
    # mkdir -p bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion
  5. Place the Hazelcast client configuration file named 
    hazelcast-client.xml
    in the ExternalHazelcastSharedStateProviderAssertion directory. The name is case sensitive.
  6. Run this command to set proper owner of the directory and files:
    # chown -R layer7:gateway /opt/SecureSpan/Gateway/node/default/etc/bootstrap
  7. Run this command to set the appropriate permission for the directory:
    # chmod  -R 2750 /opt/SecureSpan/Gateway/node/default/etc/bootstrap/
  8. Run this command to set the appropriate permissions for the file:
    # chmod 644 /opt/SecureSpan/Gateway/node/default/etc/bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion/hazelcast-client.xml
  9. Set the following properties to 
    externalhazelcast
     (case sensitive):
    • com.l7tech.server.extension.sharedKeyValueStoreProvider
    • com.l7tech.server.extension.sharedCounterProvider
    • com.l7tech.server.extension.sharedClusterInfoProvider
      Switching between providers will not migrate existing data to the newly configured provider.
  10. Restart the Gateway to connect to the External Hazelcast Storage.