Connect to an External Hazelcast Datastore

This topic describes how to configure the to use an external in-memory Hazelcast® data store. Such a configuration can improve performance in large data flow scenarios.
gateway10
This topic describes how to configure the
Layer7 API Gateway
to use an external in-memory Hazelcast® data store. Such a configuration can improve performance in large data flow scenarios.
The sample files provided on this page are not for production use and must be customized to your Gateway configuration.
Contents:
2
2
What is Hazelcast?
Hazelcast-IMDG-Logo-Orange_Dark_200px.png
Hazelcast is an in-memory data grid service. In a Hazelcast grid, data is distributed evenly between the nodes of a computer cluster, allowing for horizontal scaling of processing and available storage. Backups are distributed among the nodes to protect against failure of any single node. Hazelcast provides central, predictable scaling of applications through in-memory access to frequently-used data and across an elastically scalable data grid. These techniques reduce the query load on databases and improve speed.
You can configure Hazelcast to run in the following configurations:
  • On-premise
  • Cloud (Amazon Web Services, Microsoft Azure, Cloud Foundry, Google Cloud Platform, OpenShift® )
  • Virtual (VMware)
  • Docker containers
Create a Hazelcast Client Configuration File
Creating the Hazelcast client configuration file is required for all Gateway form factors. The file contains the Hazelcast server address plus additional client configuration.
The Gateway does not support all Hazelcast client configuration parameters. See Supported Hazelcast Client Configuration.
The following is an example of a client-side Hazelcast configuration file
:
<hazelcast-client xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.hazelcast.com/schema/client-config http://www.hazelcast.com/schema/client-config/hazelcast-client-config-3.10.xsd" xmlns="http://www.hazelcast.com/schema/client-config"> <instance-name>gatewayHazelcastClient</instance-name> <network> <cluster-members> <!-- Set your Hazelcast server node addresses here --> <address>hazelcast_host_ip:5701</address> </cluster-members> <connection-attempt-limit>0</connection-attempt-limit> <redo-operation>true</redo-operation> </network> <connection-strategy async-start="false" reconnect-mode="ON" /> </hazelcast-client>
Supported Hazelcast Client Configuration
The
Layer7 API Gateway
relies on the configuration within
hazelcast-client.xml
. However, the
Layer7 API Gateway
requires certain parameter values and does not support all configurations of the Hazelcast client.
Configuration elements apply to all Gateway form factors unless noted.
Configuration Parameters
Example
Description
instance-name
<instance-name>gatewayHazelcastClient</instance-name>
The name of the Hazelcast client. The instance-name is required and used by Gateway code.
connection-strategy
<client-connection-strategy async-start="false" reconnect-mode="ON" />
Two attributes in this configuration specify how a Hazelcast client connects to the Hazelcast cluster at startup and after disconnections.
The attributes must be set to the following values:
async-start
:
false
(default value).
Synchronizes the creation of the Hazelcast client with the establishment of a Hazelcast cluster connection. When async-start is set to false, synchronization is required, which ensures that a client/server Hazelcast connection exists before the Gateway starts.
reconnect-mode
:
ON
Reconnection attempts are allowed.
The ASYNC value is not supported and throws exceptions. The OFF value prevents reconnection attempts.
connection-attempt-limit
<connection-attempt-limit>0</connection-attempt-limit>
Defines the number of connection attempts the Hazelcast client makes for any connection scenario. Connection attempts include both startup and reconnection attempts if the client disconnects from the Hazelcast cluster. If a client/server connection is not established with all Hazelcast server nodes within the attempt limit, then the Hazelcast client shuts down.
Recommended value: 0
(zero is used for max number of attempts, equivalent to 2147483647)
Hazelcast documentation recommends tuning connection attempt parameters to define client behavior for Hazelcast cluster failures.
Related configuration for connection-timeout has a default of 5000 ms, which is reasonable. It represents how long a Hazelcast client waits for a server node to respond to its connection request, after which it marks the request as a failed connection attempt.
connection-timeout
<connection-timeout>5000<connection-timeout>
The amount of time a Hazelcast client waits for a server node to respond to its connection request. When the connection-timeout value is exceeded, the request is considered a failed connection attempt. Value is represented in milliseconds.
smart-routing
<smart-routing>true<smart-routing>
The default setting of
true
results in better performance.
This parameter is supported only by Container Gateways on OpenShift. The Hazelcast service and the Gateways must be in the same OpenShift environment.
Smart routing refers to the Hazelcast client being aware of all Hazelcast server nodes, and then sending its requests to the best Hazelcast node (for example, the node owning the data).
redo-operation
<redo-operation>true</redo-operation>
Hazelcast client requests can fail for any number of reasons. Read-only operations are retried by default. If set to
true
, then all operations (read-only and non-read-only) are retried.
Default:
false
near-cache
Not supported.
The
near-cache
setting configures Hazelcast clients to cache data locally on the client.
WARNING:
This configuration is not supported by the Gateway and should not be present. The Gateway will not start up if this setting is present.
Enable External Hazelcast Storage for a Container Gateway
As of Gateway version 10.0 CR2, Hazelcast configuration procedures and examples intended for Docker Engine and OpenShift deployments are under 'best-effort' support. Moving forward, Layer7 recommends that Container Gateway users adopt the Helm Chart method of deploying their Gateways and Hazelcast solutions to a Kubernetes environment.
3
3
Kubernetes with Helm Chart
The sample Helm Chart for deploying the Container Gateway to Kubernetes is currently stored and maintained in the Layer7 CAAPIM GitHub repository. Within the Helm Chart, there are sample configurations of Hazelcast.
The Hazelcast client configuration file is embedded directly into the configmap.yaml template file.
hazelcast-xml: |+ <hazelcast-client xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.hazelcast.com/schema/client-config http://www.hazelcast.com/schema/client-config/hazelcast-client-config-3.10.xsd" xmlns="http://www.hazelcast.com/schema/client-config"> <instance-name>{{ .Release.Name }}-{{ .Release.Namespace }}</instance-name> <network> <cluster-members> {{ if .Values.hazelcast.external }} <address>{{ required "Please set an external Hazelcast URL in values.yaml" .Values.hazelcast.url }}</address> {{ else }} <address>{{ .Release.Name }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.hazelcast.service.port }}</address> {{ end }} </cluster-members> <connection-attempt-limit>10</connection-attempt-limit> <redo-operation>true</redo-operation> </network> <connection-strategy async-start="false" reconnect-mode="ON" /> </hazelcast-client>
The client configuration file is also mounted to the volumes section of the deployment.yaml template file:
{{ if or (.Values.hazelcast.enabled) (.Values.hazelcast.external) }}- name: {{ template "gateway.fullname" . }}-hazelcast-clientmountPath: /opt/SecureSpan/Gateway/node/default/etc/bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion/hazelcast-client.xmlsubPath: hazelcast-client.xml{{ end }}
The EXTRA_JAVA_ARGS environment variables are configured for the external Hazelcast in the configmap.yaml file:
EXTRA_JAVA_ARGS: {{ template "gateway.javaArgs" . }} -Dcom.l7tech.server.extension.sharedCounterProvider=externalhazelcast -Dcom.l7tech.server.extension.sharedKeyValueStoreProvider=externalhazelcast -Dcom.l7tech.server.extension.sharedClusterInfoProvider=externalhazelcast
When deployed to Kubernetes, these template files and environmental variables (among others) are evaluated by Helm as part of the Gateway chart.
Ensure that Hazelcast is enabled in the values.yaml file by setting the Hazelcast 'enabled' value to true.
To install the Gateway Helm Chart with the external Hazelcast instance:
$ helm install -f ./ssg-hazelcast-service-metrics.yaml <release_name> --set-file "ssg.license.value=./LICENSE.xml" --set "ssg.license.accept=true"
This Helm command effectively deploys the Gateway to Kubernetes with the Hazelcast instance enabled.
Docker Engine Environment
The following instructions describe how to enable Hazelcast storage for a Container Gateway running on Docker Engine.
To enable Hazelcast storage (Docker Engine Environment):
  1. Create the hazelcast-client.xml file locally. See Create a Hazelcast Client Configuration File.
  2. Mount the Hazelcast Client Configuration file to the volumes section inside the docker compose file.
  3. Configure the EXTRA_JAVA_ARGS environment variable to include these entries (line breaks added here for readability):
    EXTRA_JAVA_ARGS:
    "-Dcom.l7tech.server.extension.sharedCounterProvider=externalhazelcast
    -Dcom.l7tech.server.extension.sharedKeyValueStoreProvider=externalhazelcast
    -Dcom.l7tech.server.extension.sharedClusterInfoProvider=externalhazelcast"
The resulting compose file:
version: '2.2' services: api-gateway: image: caapim/gateway ports: - "8080" - "8443" - "9443" volumes: - ./hazelcast/hazelcast-client.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion/hazelcast-client.xml environment: ACCEPT_LICENSE: "false" EXTRA_JAVA_ARGS: "-Dcom.l7tech.server.extension.sharedCounterProvider=externalhazelcast -Dcom.l7tech.server.extension.sharedKeyValueStoreProvider=externalhazelcast -Dcom.l7tech.server.extension.sharedClusterInfoProvider=externalhazelcast"
Copy and paste the example file to create your own
docker-compose.yml
file, if you do not have one already.
Always copy and paste as plain text, to avoid possible reformatting issues. It is recommended that you validate the content in your file before deploying (for example, using a validation site such as yamllint.com).
By setting the
ACCEPT_LICENSE
environment variable to “true”, you are indicating you have a valid and existing commercial license for
Layer7 API Gateway
and you have reviewed and accepted the terms of the CA End User License Agreement (EULA), which governs your use of
Layer7 API Gateway
.
OpenShift Environment
The following instructions describe how to enable Hazelcast storage for a Container Gateway running on an OpenShift environment.
The OpenShift environment has very strict rules for external document mounting. When deploying the container Gateway on an OpenShift environment with external Hazelcast storage, the container image must be repackaged with the hazelcast-client.xml.
To enable Hazelcast storage (OpenShift Environment):
  1. Create the hazelcast-client.xml file locally. See Create a Hazelcast Client Configuration File.
  2. Mount the Hazelcast Client Configuration file to the defined location to enable external Hazelcast storage:
    FROM image:latest
    USER root
    COPY "hazelcast-client.xml" "/opt/SecureSpan/Gateway/node/default/etc/bootstrap/assertions/ExternalHazelcastSharedStateProviderAssertion/"
  3. Push the repackaged container Gateway to the cloud repository accessible by OpenShift.
    For example:
    docker tag <customized gateway image name> <registry-host>/<image-path>
    docker push <registry-host>/<image-path>