Known Issues

This topic describes known issues in version 10.1 of the gateway, with workarounds if available.
This topic describes known issues in version 10.1 of the
Layer7 API Gateway
, with workarounds if available.
Deprecated JSON Schema v2 Still Appears in Policy Manager
: Support for JSON Schema v2 was deprecated and removed in Gateway version 10.0 CR1 because of critical CVEs in a 3rd party library. The Policy Manager, however, continues to allow users to select v2 as a schema version despite this deprecation. (DE515093)
: Users are advised to continue upgrading any existing JSON schemas from v2 to v4 in their affected Gateway policy assertions (e.g., Validate JSON Schema Assertion (Threat Protection)) before upgrading to version Gateway 10.0 CR1 or newer. Error Occurs When SSL Debug is Enabled
: Users may see the following stack trace when SSL debug logging is enabled on the Gateway:|SSLSocket duplex close failed ( "throwable" : { Socket is closed at at Source) at at at at at at Source) at com.l7tech.server.tomcat.c.close(Unknown Source) at$ at java.util.concurrent.ThreadPoolExecutor.runWorker( at java.util.concurrent.ThreadPoolExecutor$ at org.apache.tomcat.util.threads.TaskThread$ at} )
This issue is caused by the Java Development Kit (JDK), as documented here, and occurs in Java 11. (DE494717)
: None. A fix will not occur until the Gateway moves to the next long-term release of JDK after Java 11.
Browser-Based Gateway Backups No Longer an Option on the Policy Manager
: The option to use browser-based administration to perform Gateway backups was removed from the Policy Manager due to the deprecation of the Policy Manager browser client. (DE509146)
: If a browser or web-based method is required for a Gateway backup, users are advised to use RESTMAN to enable this option.
To enable browser-based administration via RESTMAN:
Determine whether an existing HTTPS port (8443 or 9443) already has browser-based administration enabled with the following command:
https://<Gateway Hostname>:8443/restman/1.0/listenPorts GET
If enabled, you should see a 'Browser-based administration' line as shown in the following sample output:
<l7:EnabledFeatures> <l7:StringValue>Published service message input</l7:StringValue> <l7:StringValue>Administrative access</l7:StringValue> <l7:StringValue>Browser-based administration</l7:StringValue> <l7:StringValue>Built-in services</l7:StringValue> </l7:EnabledFeatures>
If it is not included in the HTTPS port, you will need to add the following line to the 'EnabledFeature' section of the listenPort bundle XML:
<l7:StringValue>Browser-based administration</l7:StringValue>
After editing the bundle XML, submit the edited bundle to update the listenPort entity. Once enabled, you may use the browser-based backup method.
Luna HSM: Patch Required for Importing Private Keys Generated by OpenSSL with PSS 256
: Applies to Luna HSM Gateway users with version 10.2 of the Luna client installed. Users may see a '' error message when attempting to import private keys generated by OpenSSL with PSS 256. (DE504253)
: If the ability is to import a key generated by OpenSSL with PSS256 is required, Luna HSM users must contact Thales support for instructions to install a Luna patch for a fix.
TLS 1.3 Post-Handshake Authentication Not Supported
: TLS 1.3 post-handshake authentication is not supported by the Gateway for both inbound and outbound traffic due to a documented JDK limitation. (DE504887)
: Because there are no future plans for JDK to support post-handshake authentication, Gateway users are advised to either disable post-handshake authentication from the client/back-end side OR use TLS 1.2 instead for Gateway message routing.
Obsolete Elliptical Curve (EC) Key Support Phased Out from Gateway Message Routing
: Certain EC keys deemed as obsolete or weak by Java can no longer be used for Gateway message routing purposes. For example, the following EC keys are now considered obsolete:
  • sect163k1
  • sect193r1
  • k-163
  • p-224
Attempting to use these keys may yield an "Unable to obtain HTTP response" error. (DE507971)
: Users are advised to choose current/stronger EC keys instead. You can learn more about deprecated/obsolete EC keys via OpenJDK documentation here:
Liquibase Database Exception Causing Database Schema Upgrade Failure
: Some users are reporting the following error when attempting to upgrade their MySQL database schema as part of a Gateway upgrade (DE490936):
Migration failed for change set upgrade_x.y.z.xml::create_published_service_properties::gateway: Reason: liquibase.exception.DatabaseException: Error executing SQL CREATE TABLE ssg_testUpgrade.published_service_property (published_service_goid BINARY(16) NOT NULL, name VARCHAR(128) NOT NULL, value MEDIUMTEXT NOT NULL): Table 'published_service_property' already exists
: The workaround consists of several steps. You will compare the DATABASECHANGELOG entries with entries belonging to another baseline Gateway of the same version. If there isn't another existing Gateway database available to serve as a baseline for comparison, you'll need to install a new Gateway in a different server (separate from the production server) for this purpose. For example, if you’re upgrading from version 9.4 to 10.0 of the Gateway, perform a fresh installation of Gateway version 9.4 in a separate server. Next, compare the DATABASECHANGELOG table from the freshly installed SSG Gateway version 9.4 database with the one from the Gateway that is experiencing upgrade difficulties:
On both Gateway MySQL databases, run:
# use ssg; # select count(*) from DATABASECHANGELOG;
If the returned count numbers differ between the two databases, it’s likely the DATABASECHANGELOG was corrupted in the problematic/production Gateway.
You’ll want to rectify this by copying the change log entries from the database of a baseline Gateway or freshly installed Gateway and replacing the log entries from the database of the problematic Gateway:
  1. Backup the production database which contains the Gateway schema you wish to upgrade.
  2. On the baseline or fresh installation of the Gateway of the same version, run the following command:
    # mysqldump ssg DATABASECHANGELOG > /home/ssgconfig/dbchangelog_rows.sql
  3. Open
    and locate the following line:
  4. Copy the entire line including the contents inside (...).
  5. Stop the production Gateway before making changes to the database:
    # service ssg stop
  6. From the MySQL client, run:
    # use ssg; # delete * from DATABASECHANGELOG;
  7. Paste the
    line into the command line to run.
  8. Run the following command to save your changes to the production database that you are upgrading:
    # commit;
  9. Restart the production Gateway with the following command:
    # service ssg start
  10. Continue to upgrade the production database schema using the SSG Config Menu as before.
Manage Keystore - Unable to Set Preference to Use the SafeNet HSM For Cryptographic Operations
: Affects Luna SafeNet HSM customers who have installed Luna client version 10.2 on their machines running Gateway version 10.0 CR 3. When enabling or connecting to the SafeNet HSM with the Manage Keystore function of the Policy Manager, the 'Prefer to use this device for all cryptographic operations' check box cannot be selected as an option. The purpose of selecting this option is to ensure that all cryptographic operations are performed on the Luna HSM appliance itself.
: None.
nShield HSM - Cannot Access Policy Manager When Level 3 (Strict) FIPS Mode is Enabled
: Impacts nShield HSM users (running nShield client version 12.60.x or newer with Gateway version 10.0 CR3 or newer installed) who require their HSM to operate in FIPS 140-2 Level 3 mode. Users are unable to log into the Policy Manager due to a SSLHandshakeException error caused by a known Java defect. (DE498599)
: Users requiring to run their nShield HSM in FIPS 140-2 level 3 should NOT upgrade their Gateway version to 10.0 CR3 or newer until further notice.
System Reboot Issues
Gateway node sometimes takes a longer time to boot up after applying a patch.
To improve system reboot time, follow these steps for each Gateway node:
  1. Take a snapshot of your VM.
  2. Run the following commands in the same order as mentioned below:
    # cd /etc/rc.d/init.d
    # ./layer7-run-once-on-boot
    This command might take a few minutes to execute.
    # reboot -n
Your Gateway node boots up quickly. Repeat this task for every node in the cluster.
Gateway Now Uses SunJSSE TLS Provider: Increased Time for Authentication Bind Operation
Impacts all Gateway form factors that perform LDAP authentication. Beginning with Gateway version 9.2, SunJSSE became the primary TLS provider, which is used by LDAP user authentication when the Gateway communicates with an LDAP provider over a secure network. Unlike previous TLS providers (i.e., the deprecated FIPS/BSAFEJSSE), LDAP binds performed by SunJSSE require a full TLS handshake for each LDAP user. This process adds approximately
100 ms to the LDAP authentication bind for each user
when compared to the same process for Gateway version 9.1. (DE474013)
The Gateway provides authorization caching of previously completed or successful binds.To reduce the performance impact of the full TLS handshake for repeat binds against the same user, the authorization cache size and storage of successful binds may be increased:
  • authCache.successCacheSize (default is 2000)
  • authCache.maxSuccessTime (default is 60000 ms)
Both properties can be configured as Gateway Cluster Wide Properties - see Credential Caching Cluster Properties for more information.
The optimal size of the cache is dependent on the size of your LDAP user base. Note that the cache success will not take into account of users changing passwords or being disabled on the LDAP server while success is cached.
Connection Error in Connect To Outbound WebSocket Assertion
During the end-to-end connection upgrade process, and right after the outbound connection is succeeded, Gateway shows the following error message from backend (outbound): (DE467783)
com.l7tech.external.assertions.websocket.server.WebSocketOutboundHandler: Failed to send message to inbound: Message could not be delivered, connection no longer available.
You will not see this issue once the end-to-end connection upgrade is completed.
CentOS 7 Issues
Gateway version 10.0 supports the CentOS 7 platform. The following are known Gateway issues related to the CentOS 7 platform.
Update my.cnf File to Fix Issue with mysqld Generating Error Logs
In Layer7 API Gateway 10.0 OVA installation, mysql error logs are generated in
instead of
as defined in the
In CentOS 7, systemd manages MySQL server startup and shutdown.
is not installed as it is not needed. Some parameters are defined in the
section in
file such as log file path and pid-file. So it might seem like mysqld is not generating error logs.
Edit the
file and move all the content from
section to
section. Restart the MySQL server.
Ensure that you overwrite the
parameter line. If there are multiple entries in the
section for a parameter, then only the last line is prioritized and earlier entries are ignored.
AMI Gateways: Network Service Showing a 'Failed' Status
AMI Gateway (version 10.0) customers may report seeing a failed status for their network service when viewing the summary of network devices and their connection status:
Dec 04 10:23:44 ip-111-11-11-111.broadcom.internal network[967]: Bringing up interface eth0: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device eth0 does not seem to be present, delaying initialization. Dec 04 10:23:44 ip-111-11-11-111.broadcom.internal network[967]: [FAILED] Dec 04 10:23:44 ip-111-11-11-111.broadcom.internal network[967]: Bringing up interface ssg_eth0: [ OK ]
None required. Due to the predictable network interface naming scheme introduced by CentOS 7, the AMI network service will first look for "eth0" when it should be looking for "ssg_eth0", the new name that was given by Gateway. As long as the service is able to verify the existence of "ssg_eth0" with an OK status, no actual system failure will occur despite the initial FAILED status associated with "eth0". (DE441402)
Cloned OVAs Require Manual Patching
After cloning a Gateway OVA (version 10.0), the Gateway administrator is unable to SSH into it. Newly deployed clones will have different network adapter hardware addresses, causing the VM to create their own unique network interface names with the predictable device naming scheme. Because of the name conflict between the original OVA and the clone, the firewall rules in the iptables prevents the user from accessing OVA clones via SSH. This issue is known for the Gateway OVA that runs on CentOS 7. (DE441413)
Customers planning to clone Gateway OVAs should do the following:
  • Disable iptables/ip6tables services prior to cloning.
  • After cloning the OVAs, ensure that the interfaces identified in the network configuration files (i.e., ifcg files) are aligned with the interface names that appear in the iptables/ip6tables. You may use the
    command as part of your reconfiguration. See the following procedure as an example.
To change a device name from ens160 to ssg_eth0 on a cloned OVA
  1. Log into the cloned OVA as a root user in console.
  2. List the available connections to the Gateway with the following command:
    nmcli connection show
    Two connections should be displayed:
    • One connection with a NAME of 'ssg_eth0' without a DEVICE assigned.
    • Another connection with a NAME of 'Wired connection 1' with DEVICE 'ens160' assigned. Your machine/setup may use a different DEVICE name.
  3. Delete the connection without a DEVICE assigned (i.e., ssg_eth0) with the following command:
    nmcli connection delete ssg_eth0
    You may verify the update with the
    nmcli connection show
  4. Update the connection with the ens160 DEVICE assigned with the following command (use the appropriate DEVICE name that suits your machine and setup):
    nmcli connection modify "Wired connection 1" 802-3-ethernet.mac-address "$(cat /sys/class/net/ens160/address)" con-name "ssg_eth0" ifname "ssg_eth0"
    The connection with the NAME 'Wired connection 1' and assigned DEVICE of 'ens160' is renamed to NAME 'ssg_eth0', but DEVICE is still assigned as 'ens160'
    You may verify the update with the
    nmcli connection show
  5. Reboot the OVA with the
    command and log back into the cloned OVA again as a root user.
  6. List the latest connections with the
    nmcli connection show
    command to verify that the connection with the 'ssg_eth0' NAME is now assigned with the 'ssg_eth0' DEVICE.
  7. Turn on iptables.
  8. You should now be able to successfully SSH into the cloned OVA.
The Exited Status: What It Means for Gateway-Related System Services
Previously, in CentOS/RHEL 6, init scripts were used to start and stop services. Under systemd, these scripts have been replaced with systemd service units; more importantly, the newly introduced 'exited' status is unique to systemd and may cause confusion if you are new to CentOS 7. For example, if you are checking the status of a service and want to know if it's running, you may come across the
Active: active (exited);
status, as opposed to
Active: active (running)
. This usually means that systemd executed the commands specified in the service unit file but cannot verify if the process is actually running because an associated daemon that monitors that process was not found. Because Gateway does not use a daemon, you may see this status when checking or running services in CentOS 7. (DE437214)
None required.
Policy Manager Issues
Policy Manager Startup Takes A Long Time for New User
When a new user is created and assigned various different roles, it may take up to 30 minutes for the User to log in to the Policy Manager, compared to the usual startup time for an Admin User (around a couple of minutes). (DE222762) (DE413957)
The following procedure can be used to reduce the roles items of a user, which may improve the speed and performance of permission validation:
  1. Create a new security zone. Two new roles will become available:
    • View X Zone
    • Manage X Zone
  2. Assign all the entities which should be available for a group of users (for example, developers) to this security zone.
  3. Do one of the following:
    • Only assign View/Manage X Zone roles to the user role.
    • Create a new group, assign View/Manage X Zone roles as well as other necessary predefined roles to this group. Only assign this new group to the user role.
  4. Disable auto-assign roles by setting these cluster-wide properties, to ensure the number of roles for a user does not automatically increase:
Upgrading Gateway in Azure with an Azure MySQL Database Causes Syntax Error
: Some customers report receiving the following error when upgrading their Gateway with an Azure MySQL database: 
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '@'localhost'' at line 1.
  1. Backup your Azure MySQL database.
  2. In privileged shell, enter the following command:
    /opt/SecureSpan/JDK/bin/java -jar /opt/SecureSpan/Gateway/runtime/lib/liquibase-3.2.2.jar --driver=com.mysql.jdbc.Driver --classpath=/opt/SecureSpan/Gateway/runtime/lib/mysql-connector-java-5.1.46.jar --changeLogFile=/opt/SecureSpan/Gateway/config/etc/db/ssg.xml --url=jdbc:mysql:// --username='[email protected]' --password='x' update
    A 'Liquibase Update Successful' message should be returned.
  3. To confirm that the Gateway version has been upgraded to 10.1, enter the following in MySQL:
    select * from ssg_version
    If version 10.1.00 is not returned for current_version, then enter the following:
    update ssg_version set current_version = '10.1.00';
Cloud Foundry Installations of Container Gateway Show Error logs at Startup
: During Gateway startup, logs regarding the initial configurations of Gateway are shown as ERR on Cloud Foundry. (DE359636)
Connection Timeout During SSH Routing
When the "SFTP" protocol is selected in the Route via SSH2 Assertion, the connection timeout is twice as long as the value specified in the assertion. (DE215650)
Reduce the timeout to one half the desired value.
Context Variable ${request.time} Returns Incorrect Time When Used in Encapsulated Assertions
is used in the underlying ("backing") policy of an encapsulated assertion, it returns an incorrect value.
Context Variables with ${service.*} Prefix Have a Global Scope
Context variable that have the prefix "service" results in a globally scoped variable, instead of the local scope as expected. (DE293047)
  • The value for context variable ${service.AAA} set in a global policy can be retrieved from the service policy. The expected result is that the variable is empty when access from the service policy.
  • The value for context variable ${service.BBB} set in the underlying policy fragment for an encapsulated assertion can be retrieved from the service policy. Assuming the variable is not configured to be returned as a parameter, the variable should be empty when accessed from the service policy.
Use a prefix other than "service" if you do not want the context variable to have a global scope.
Dependency Errors When Deleting OAuth Folder
Deleting an OAuth policy that contains encapsulated assertions trigger dependency errors in the Policy Manager. These errors cannot be acknowledged in bulk nor can the deletion be canceled. (DE222685)
To avoid this, see "Problem: Deleting OAuth policy creates dependency errors" in Troubleshoot: Problems and Solutions.
Exceptions Being Thrown by Gateway When 'Sending Reply to Specified Queue' is Used
: Gateway is throwing exceptions when using the 'Sending Reply to Specified Queue'  Inbound option for MQ 8.0. (DE386344)
False Warning Messages in Gateway Log
The Gateway log contains a warning similar to the following:
WARNING 1 com.l7tech.server.policy.PolicyCacheImpl: 3255: Policy "Google Auth Code Extension" (#d095d9a4c56c975b34c64db5a6741dd0) contains an unlicensed assertion: Unknown assertion: RetrieveOAuth2Token
The RetrieveOAuth2Token assertion is part of the OAuth Toolkit, which requires no special license. Only the standard Gateway license is required.  (DE317426)
None required. The RetrieveOAuth2Token assertion is indeed present and functioning normally. The erroneous log message is caused by internal timing issues during the license check stage of Gateway startup. You may safely ignore this log message.
File Permissions After a Restore
Some file permissions may not be correctly restored after a Gateway is restored using the
command. This results in the Policy Manager being unable to connect to the Gateway. (DE221036)
See "Problem: Gateway is not running properly after a restore" in Troubleshoot the Gateway.
JSON Path Assertion Returns Error With Null Value
: When evaluating a JSON expression with a value of
(case-sensitive), the assertion fails. (DE384413)
: Use the JSON Path Expression V2 assertion.
Key Generation Fails with Luna HSM
When Private Key Properties). (DE305070, DE314879)
Use a key type other than the "Elliptic Curve" key types.
Manage Roles Performance Issues
Performance and speed of Manage Roles may be impacted when many (> 500) custom roles are created, and each role has numerous (200 to 300) permissions. (DE326081)
Avoid excessive number of roles and permissions.
Migration Issue Between Gateways
Migration issues may occur after upgrading. Release 9.1 introduces a new "gateway-hazelcast" FIREWALL_RULE. This rule is automatically added to the Gateway upon restart, if it does not already exist. This entity is created with a unique ID per Gateway. If you attempt a bundle import, the mapping for the "gateway-hazelcast" FIREWALL_RULE must be "MapBy:name" to prevent a 409 conflict. (DE211367)
Modify the bundle file that is created during bundle export as follows:
Locate this line:
<l7:Mapping action="NewOrExisting" srcId="
" srcUri="http://
" type="FIREWALL_RULE"/>
And then modify it as follows:
<l7:Mapping action="NewOrExisting" srcId="
" srcUri="http://
<l7:Property key="
Multi-Valued Variable Error with Look Up Item by Value Assertion
The Look Up Item by Value Assertion fails if a multi-valued variable contains the same value multiple times. (DE305234)
For example:
Set Context Variable fave as String to: banana
Set Context Variable fruits as String to: apple,banana,pear,banana
Split variable fruits into fruitsalad on ","
Look Up Item by Value: find item ${fave} within ${fruitsalad}; output index to ${index}
Return Template Response to Requestor -- ${index}
This fails to return "1,3". However, in the ssg_0_0.log file, a similar INFO entry to the following is logged:
com.l7tech.external.assertions.xmlsec.server.ServerIndexLookupByItemAssertion: 6: More than one value was matched. Exception caught!
Use the following sample workaround policy:
To use the workaround policy:
  1. Download the .xml file to your computer.
  2. Click
    Import Policy
    on the Policy Tool Bar to import the policy into the Policy Manager. For more information, see Importing a Policy from a File.
Occasional Outbound JMS Request Processing Error
There is a known issue with NamingException where outbound JMS messages are not processed correctly and returned the following error:
com.l7tech.server.policy.assertion.ServerJmsRoutingAssertion: 4: Error in outbound JMS request processing: Something already bound at (queue-name). Exception caught!
However when the messages are reprocessed, they behave as expected. (DE304792)
Perform JDBC Query Assertion Error When Using Hyphen in Procedure Name
When setting up a Perform JDBC Query Assertion, using hyphen in the stored procedure name will fail the query, as it does not comply with JDBC's naming conventions. (DE309730)
Do not use hyphen in the procedure name.
Private Key is Created without Special Purpose When Loaded from Bootstrap/Bundle Folder or Sent to RESTman Endpoint
When trying to create a private key with a defined special purpose in a RESTman bundle via the bootstrap/bundle folder of the Container Gateway, that private key is created but without the special purpose (e.g. Default CA). (DE320421)
Published Service Properties Dialog Opens Slowly Using Java Web Start Policy Manager
When using the Java Web Start version of the Policy Manager Published Service Properties dialog for the first time takes much longer than expected. (DE326621)
None. The browser client needs to download additional .jar files upon first invocation of the Published Service Properties dialog. Subsequent opening returns to normal speed.
Retrieving Certificate When Gateway in FIPS Mode
The Add Certificate Wizard fails to retrieve a third-party certificate using the “Retrieve via SSL Connection (HTTPS or LDAPS URL)" option if the Gateway is in FIPS mode (cluster property
= true). (DE211158)
  1. Retrieve or download the certificate file using external means. For example, you can use this OpenSSL command:
    # echo | openssl s_client -connect 2>/dev/null | openssl x509 -outform PEM > cert.pem
  2. Import the certificate to the Gateway by navigating to
    Tasks > Certificates, Keys, and Secrets > Manage Certificates > Add > Import from a file
Route via HTTP(S) Assertion Unexpected Failure
The [Other] tab in the HTTP(S) Routing Properties contains the control "Never fail as long as target returns an answer". There is a known issue that can still cause the assertion to fail even when an answer is returned under the following scenario:
  • In the [Security] tab, the Service Authentication is “Use HTTP Credentials from Request”.
  • The backend services return a 401 HTTP error. (DE215522)
To prevent assertion failure in this scenario, use “Specify HTTP Credentials” as the Service Authentication method instead. Specify the username and password as context variables from the request (for example,
Siteminder Cluster-Wide Property Validation Warning
The cluster-wide property validation type for the siteminder.session.generateCookieString was set incorrectly and you will see this WARNING message when Gateway starts. "WARNING 973 com.l7tech.util.ConfigFactory$DefaultConfig: Ignoring unknown type '(?i)true|false' for validation of property ‘siteminder.session.generateCookieString’". This should not cause any functionality issue.