Access Gateway Troubleshooting

The following content details the smsps troubleshooting information:
casso1283
The following content details the
Access Gateway
troubleshooting information:
Fix for the CVE-2020-9484 Vulnerability in Apache Tomcat
CVE-2020-9484 in Apache Tomcat allows an attacker execute remote code execution through deserialization of the file under their control under specific conditions. To fix the vulnerability in Release 12.8.02 or 12.8.03, we recommend that you apply the latest SiteMinder patch related to this vulnerability on Access Gateway in your environment.
Follow these steps
:
  1. Download the solution for the CVE-2020-9484 vulnerability from the SiteMinder Cumulative Release Index page. The solution contains the following files:
    • bin.zip
    • conf.zip
    • lib.zip
    • ROOT.zip
    Unzip the files.
  2. Stop Access Gateway.
  3. Take a backup of the
    bin
    ,
    conf
    ,
    lib
    , and
    webapps
    folders at
    access_gateway_installation_path
    /secure-proxy/Tomcat
    location.
  4. Copy all the files in the downloaded
    bin
    folder into
    access_gateway_installation_path
    /secure-proxy/Tomcat/bin
    folder.
  5. Copy all the files in the downloaded
    conf
    folder into
    access_gateway_installation_path
    /secure-proxy/Tomcat/conf
    folder.
  6. Copy all the files in the downloaded
    lib
    folder into
    access_gateway_installation_path
    /secure-proxy/Tomcat/lib
    folder.
  7. Copy all the files in the downloaded
    ROOT
    folder into
    access_gateway_installation_path
    /secure-proxy/Tomcat/webapps
    folder.
  8. Start Access Gateway.
Fix for the CVE-2020-1938 Vulnerability in Apache Tomcat
The CVE-2020-1938 vulnerability (Ghostcat vulnerability) in Apache Tomcat targets its AJP when it is externally exposed. To fix the vulnerability in Release 12.8 through 12.8.03, we recommend that you apply the latest SiteMinder patch related to this vulnerability on Access Gateway in your environment. From Release 12.8.04, the fix is available by default in SiteMinder.
For information about the vulnerability, see Apache Tomcat documentation.
Follow these steps for Release 12.8 through 12.8.03
:
  1. Download the solution for the Ghostcat vulnerability from the SiteMinder Cumulative Release Index page.
    The solution contains the following files:
    • proxyrt.jar
    • annotations-api.jar
    • catalina.jar
    • catalina-ant.jar
    • catalina-ha.jar
    • catalina-tribes.jar
    • ecj-4.4.2.jar
    • el-api.jar
    • jasper.jar
    • jasper-el.jar
    • jsp-api.jar
    • servlet-api.jar
    • tomcat-api.jar
    • tomcat-coyote.jar
    • tomcat-dbcp.jar
    • tomcat-i18n-es.jar
    • tomcat-i18n-fr.jar
    • tomcat-i18n-ja.jar
    • tomcat-i18n-ru.jar
    • tomcat-jdbc.jar
    • tomcat-util.jar
  2. Stop Access Gateway.
  3. Copy the files within the solution that you downloaded in
    Step 1
    into the
    access_gateway_installation_path
    /secure-proxy/Tomcat/lib
    location.
    Important!
    Take a backup of all the files before you replace them.
  4. Open the server.conf file.
  5. Add the following lines of code in the
    General Server Information
    section of the file:
    ajp13.secretRequired=true worker.ajp13.secret=
    ajp_secret
    ajp_secret
    defines the shared secret between Tomcat and Access Gateway to prevent unauthorized connections using AJP. The value must be specified without any quotes.
  6. (
    Additional step for IPv6 environment
    ) Set
    worker.ajp13.host
    to
    ::1
    .
    worker.ajp13.host=::1
  7. Save the changes.
  8. Start Access Gateway.
server.conf Does Not Contain Changes of CVE-2020-1938 Vulnerability Fix
From Release 12.8.04, SiteMinder includes the fix for the CVE-202-1938 vulnerability (Ghostcat vulnerability) in Apache Tomcat. However, when an earlier version is upgraded to 12.8.04, the server.conf file does not contain the parameters that are related to the fix.
To resolve the issue, perform the following steps:
  1. Open server.conf file.
  2. Add the following lines of code in the
    General Server Information
    section of the file:
    ajp13.secretRequired=true worker.ajp13.secret=
    ajp_secret
    ajp_secret
    defines the shared secret between Tomcat and the reverse proxy server in front of Tomcat to prevent unauthorized connections using AJP. The value must be specified without any quotes.
  3. Save the changes.
  4. Restart Access Gateway.
Access Gateway on Windows Crashes When NTLM Authentication Goes Through Load Balancers
Symptom
:
The authentication flow requires sticky-bit load balancing to ensure that all requests of a user goes to the same Access Gateway server once the session is established. However, if your Access Gateway is on Windows, the load balancer occasionally does not honor the sticky bit configuration, which is required for NTLM authentication. This issue sometimes might crash the SSPICLI.DLL of the server, thus crashing Access Gateway.
Solution:
Create a new ACO parameter named 'UseNtlmMapforNtlmAuth' and set the value to yes. This parameter avoids the crash and returns an appropriate failure message when there is a problem.
Secure Log4j Logs
You can use Log4j pattern layouts to secure the Log4j logs in your environment. Configure the following conversion pattern in the Log4j configuration file:
enc{pattern}{[HTML|XML|JSON|CRLF]}
For detailed information about Log4j pattern layouts, see Apache Log4j documentation.
Access Gateway Log Files
Two log files located in ag_home\secure-proxy\proxy-engine\logs provide important information about Access Gateway:
  • proxyui.log – Logs the information related to ProxyUI.
  • nohup.outYYYYMMDD_hhmmss.log – Contains all STDOUT and STDERR messages in applications or loggers deployed in Tomcat.
Session Creation Fails in SSL ID Session Scheme
Symptom
:
If you are using SSL ID Session Scheme with SSL session tickets, you may receive the following error message when you access a resource:
session create error
Solution
:
SSL ID Session Scheme is not supported with SSL session tickets. To disable the support for SSL session tickets, perform the following steps:
  1. Navigate to the accessgateway_home/httpd/conf/extra/ location.
  2. Open the httpd-ssl.conf file and navigate to the SSLSessionTickets directive.
  3. Set SSLSessionTickets to off.
  4. Save the changes.
A Pop-up Window Appears in the Browser after SSL Configuration
Symptom:
When I access
Access Gateway
using an internet browser after SSL configuration, a pop-up window is displayed.
Solution:
If you use an internet browser with weak encryption algorithm to access
Access Gateway
for the first time after configuring SSL, a pop-up window is displayed. You can allow the pop-up and can continue to access
Access Gateway
. If you do not want the pop-up to display, perform one of the following steps:
  • Use strong internet browsers such as Internet Explorer 8.0, Chrome, Mozilla Firefox that support strong encryption algorithms.
  • If you are not using an X509 based authentication scheme, perform the following steps:
    1. Navigate to <install_path>/httpd\conf\extra\.
    2. Open the httpd-ssl.conf file.
    3. Navigate to SSLVerifyClient and set the value to none.
    4. Save the change.
    5. Restart <productname>.
Unable to Start Apache on UNIX systems
Symptom
:
When running the
Access Gateway
on a UNIX system, the Apache server fails to start. In the Apache log file, the following error message appears:
Invalid argument: setgid: unable to set group id to ...
Solution
:
This error occurs when the group for the Run-As-User on UNIX systems does not correspond to the group specified in the Apache configuration file (httpd.conf). If you see this error, edit the Group directive in the Apache httpd.conf file. To edit the Group directive, perform the following steps:
  1. Remove the comment sign (#) before the Group directive.
  2. Specify the group to which the Run-As-User belongs.
  3. Run
    Access Gateway
    startup command again (sps-ctl start or startssl).
Non-English Input Characters Contain Junk Characters
Valid on UNIX/Linux
Symptom
:
Some non-English input characters are not displayed correctly in the console window.
Solution
:
Verify the terminal settings of your console window. Confirm that the console does not clear high (eight) bit of input characters. Execute the following command:
stty –istrip
Unable to Log Federation Web Services Errors
Symptom
:
The Federation Web Services errors are not logged.
Solution
:
To log the errors in Federation Web Services, enable the AffWebServices and FWSTrace logs parameters in the LoggerConfig.properties file.
Follow these steps
:
  1. Open the LoggerConfig.properties file.
    Default Path: ag_home/secure-proxy/Tomcat/webapps/affwebservices/WEB-INF/classes/LoggerConfig.properties
  2. Configure the following parameters:
    LoggingOn=Y
    TracingOn=Y
  3. Save the changes.
DNS is Cached for Every Request
Symptom
:
I do not want
Access Gateway
to cache the DNS name look-up settings of the server.
Solution
:
The
Access Gateway
is configured by default to cache the DNS settings of the server. To change this default behavior, adjust the networkaddress.ttl setting in the java.security file.
Follow these steps
:
  1. Navigate to the directory NETE_SPS_JAVA_HOME\jre\lib\security.
  2. Open the java.security file.
  3. Set the networkaddress.cache.ttl parameter to a positive integer. For example, networkaddress.cache.ttl=2
 
networkaddress.ttl
Specifies the duration, in seconds, for which the
Access Gateway
caches the successful DNS name look-ups. Enter a positive integer. If you enter a negative value, the
Access Gateway
caches the DNS settings.
Default
: -1
Resource Request Fails
Symptom
:
Access Gateway
failed to serve a resource request.
Solution
:
To troubleshoot an error, verify the following log files for the error details:
  • spsagent and spsagenttrace logs
  • Apache access and error logs
  • httpclient.log
  • server.log
  • mod_jk.log
If the log files do not contain logs, ensure that you have enabled logging in the log files.
Configure Agent Logs
Access Gateway
logs errors that are related to the proxy engine in the agent logs. A local configuration file or an ACO in the Policy Server contains the parameters that enable error logging and determine logging options.
Follow these steps
:
  1. Open the ACO of
    Access Gateway
    in Policy Server.
  2. Set the value of the LogFile parameter to yes.
    Note
    : Setting the value of this parameter to yes in a local configuration file overrides any of the logging settings that are defined on the Policy Server.
  3. Complete the following parameters:
    • LogFileName
      Specifies the full path including the file name, to the log file.
    • LogAppend
      Adds new log information to the end of an existing log file. When this parameter is set to no, the entire log file is rewritten each time logging is invoked.
    • LogFileSize
      Specifies the size limit of the log file in megabytes. When the current log file reaches this limit, a new log file is created.
    • LogLocalTime
      Specifies whether the logs use Greenwich Mean Time (GMT) or local time. To use GMT, change this setting to no. If this parameter does not exist, the default setting is used.
  4. Restart
    Access Gateway
    .
Configure Trace Logs
You can configure the trace logs to control the size and format of the file. After trace logging is configured, you determine the content of the trace log file separately. This lets you change the types of information contained in your trace log at any time, without changing the parameters of the trace log file itself.
Follow these steps
:
  1. Locate the SecureProxyTrace.conf file, and duplicate the file.
  2. Open your Agent Configuration Object or local configuration file.
  3. Set the TraceFile parameter to yes.
    Note: Setting the value of this parameter to yes in a local configuration file overrides any of the logging settings that are defined on the Policy Server.
  4. Configure the following parameters:
    • TraceFileName
      Specifies the full path to the trace log file.
    • TraceConfigFile
      Specifies the location of the SecureProxyTrace.conf configuration file that determines which components and events to monitor.
    • TraceAppend
      Specifies if the new logging information must be added to the end of an existing log file instead of rewriting the entire file each time logging is invoked.
    • TraceFormat
      Specifies how the trace file displays the messages.
    • TraceDelimiter
      Specifies a custom character that separates the fields in the trace file.
    • TraceFileSize
      Specifies the maximum size of a trace file in megabytes.
      Access Gateway
      creates a new file when this limit is reached.
    • LogLocalTime
      Specifies whether the logs use Greenwich Mean Time (GMT) or local time. To use GMT, change this setting to no. If this parameter does not exist, the default setting is used.
  5. Restart
    Access Gateway
    .
Configure the mod_jk.log File
Access Gateway
logs all the communication messages between Apache and the proxy engine in the mod_jk.log file. By default, logging is enabled in this file and the log file is located in ag_home\secure-proxy\httpd\logs\mod_jk.log.
Follow these steps
:
  1. Open the httpd.conf file.
    Default Path
    :
    accessgateway_installation_home
    \secure-proxy\httpd\conf
  2. Modify the available parameters as required.
    Note
    : For information about configuring the httpd.conf file and the mod_jk.log file, see the Apache documentation set.
  3. Ensure that JkRequestLogFormat is set in the %w %V %T %m %h %p %U %s format.
  4. Save the changes.
  5. Restart
    Access Gateway
    .
Configure the httpclient.log File
For debug purposes only, you can enable the httpclient.log. By default, the httpclient.log file is located in ag_home\secure-proxy\proxy-engine\logs.
Follow these steps
:
  1. Open the server.conf file.
  2. Ensure that httpclientlog is set to yes.
  3. Open the httpclientlogging.properties file.
    Default Path
    :
    accessgateway_installation_home
    \Tomcat\properties directory
  4. Modify the available parameters as required.
    Note
    : For information about configuring the httpclientlogging.properties file, see the Apache documentation set.
  5. Save the changes.
The Installation Program Displays Warnings
Valid on UNIX
Symptom
:
When I install
Access Gateway
, the installation wizard displays warnings that few files must be configured manually.
Solution
:
If you do not have root permissions, you can install
Access Gateway
but the automatic installation process cannot complete all the installation steps. The installation wizard displays warnings that help you determine the files that must be configured manually.
Note
: Non-root installations are not recommended for SSL-enabled servers. A non-root installation is less secure because it allows an additional person with root permissions access to your keys and certificates.
Cannot Start the Access Gateway Server
Symptom
:
Access Gateway
server fails to start.
Solution
:
Use the following information if you cannot start your server:
  • Verify that the ServerName directive in ag_home/secure-proxy/httpd/conf/httpd.conf corresponds to the name of your server.
  • Verify that the server is not already running by executing one of the following commands:
    ps -ax|grep http on BSD compatible systems
    ps -elf|grep http on System V release 4 compatible systems
    If this results in a list of processes, stop the running server before starting your new server.
  • Check the log files in the directory ag_home/secure-proxy/httpd/logs
  • Verify that the SSLCertificateFile and the SSLCertificateKeyFile directives in the httpd.conf file point to your certificate and key files. The file is in the directory ag_home/secure-proxy/httpd/conf
  • Determine whether you are using non-IP-based virtual hosts. SSL requires IP-based virtual hosts.
  • Verify that no other server is running on the default port for the SPS. The default port is specified in the httpd.conf file.
  • If you are using SSL, be sure that you have generated a key and certificate before starting the server, otherwise you will get an error.
Cannot Access Access Gateway with a Browser
Symptom
:
Difficulty accessing the
Access Gateway
using a browser.
Solution
:
To access
Access Gateway
using a browser:
  • Verify that DNS is aware of your servername with the command nslookup servername or try to ‘ping’ your server with the ping servername command.
  • Run the server without SSL and access your web site to verify whether the problem is with the key or certificate files. To start the server without SSL, execute ./sps-ctl start in the directory ag_home\secure-proxy\proxy engine directory.
  • Try to make a telnet connection to ports 80 and 443 of your Web server (or the non-default ports you specified). If you installed as a non-root user, try to connect to ports 8080 and 8443.
Issues Configuring Virtual Hosts
Symptom
:
Difficulty configuring virtual hosts.
Solution
:
To resolve, see http://httpd.apache.org/docs-2.0/vhosts/.
Virtual Hosts Configuration Fails
Symptom
:
The configuration of virtual hosts fails.
Solution
:
For information about configuring virtual hosts, see www.apache.org.
Access Gateway is Not Forwarding Requests
Symptom
:
When I access a resource, the 404 File Not Found browser error is displayed and the action is not logged in the Web Agent log.
Solution
:
Verify the name and IP address of the virtual host in the server.conf file.
The winnt_accept: Asynchronous AcceptEx failed Error Occurs
Symptom
:
The following error occurs when Access Gateway is run:
[warn] (OS 64)The specified network name is no longer available.:
winnt_accept: Asynchronous AcceptEx failed
Solution
:
To resolve, configure the AcceptFilter parameter in the httpd.conf file. For information about the parameter, see the Apache documentation.