Enterprise Logging and Auditing of Gateway

The information in this topic applies to all forms factors of the 
Layer7 API Gateway
, not just the
Gateway - Enterprise
This topic describes how to perform enterprise logging and auditing with the 
Layer7 API Gateway
. Many organizations using the Gateway have specific requirements for logging and auditing to satisfy established service level agreements. The Gateway can provide this information at runtime about the transactions performed.
When logging data from the Gateway, consider factors such as performance and file size. For example, the Gateway performs well while gathering data about transactions, but your disk space is consumed quickly because you configured very detailed logging. The low disk space then leads to decreased performance.
CA Technologies recommends logging to a disk source off the Gateway itself. Use a queuing mechanism to process the transactional data, then manipulate the data to suit the needs of your enterprise reporting and data trending tools. This minimizes the logging overhead on the Gateway and minimizes the disk usage of the Gateway appliance’s hard drives.
Components of an Enterprise Logging/Auditing System
Enterprise logging with the 
Layer7 API Gateway
 includes the following components:
  • the Gateway cluster
  • a queue management system
  • a database
  • a data processing component
  • an enterprise reporting toolset
At a high level, this is how the components interact:
The Gateway logs messages in policy using the audit sink policy. The audit sink policy gathers all your audited information to send to another location.
The Route via MQ Native Assertion within the audit sink policy routes your logging messages to a queue. This method is efficient because no information is persisted on the Gateway for the log message and no extra processing is required by the Gateway to handle the message.
The queuing system is responsible for gathering the logging messages being pushed to the queue from the Gateway. The queuing system can move the messages from the queue to a database, feed the information to a processing application, or write the messages to a file. Using a reliable queue messaging infrastructure ensures that you will not lose the important information that you want to log.
The Gateway places all transactional information into a transactional table on the database. This information can then be read by a stored procedure and placed into an OLAP star schema. The OLAP cube can be referenced in any of the following manner:
  • data processing applications
  • enterprise logging and trending tools, which manipulate the OLAP data to generate reports and other graphical data
Moving the transactional data to a database away from the Gateway follows best practices for data warehousing scenarios, where ease of cleanup and scalability is important.
Gather Audit Data from the Gateway
The audit sink policy gathers transactional data for enterprise reporting of the Gateway's transaction information.
To enable the audit sink:
  1. In the Policy Manager, select 
    Tasks >
    Manage Log/Audit Sinks
    and then click
    Manage Audit Sink
  2. Configure the Audit Sink Properties:
    • Save audit records to Gateway database
      : This outputs audit records to the Gateway's database. Depending on the needs of your enterprise, you may disable this option. Many organizations prefer to disable this setting and have the audit logs appear only in the syslog log file. The tooling for managing the syslog logs also helps manage the file size of the syslog and rollovers and backups.
    • Output audit records via audit sink policy:
      This option enables the audit sink policy and is the best practice for enterprise logging on the Gateway. The first time that you select this option, a new "[Internal Audit Sink Policy]" is added to the Services and Policies list in the Policy Manager. This audit sink policy is run at the end, after all the other policies created to handle the Gateway requests have completed. For example, you have a Gateway service that protects a back-end Web service. The service policy runs each time a request is routed. When the service policy has finished executing, the audit sink policy is then invoked, executing the assertions within the audit sink policy. This usually involves routing the audit information elsewhere to be stored.
For more information about using the audit sink policy, see Working with the Audit Sink Policy.
Manipulate/Store the Audit Data
You can manipulate and store the data as required by your enterprise by gathering audit information from the Gateway using the audit sink policy. Depending on your infrastructure, you might have enterprise audit and logging tooling that access and use data in a specific format. After you gather the Gateway processing information and add it the queue, you can gather it off the queue and manipulate it as necessary.
You can take the data from the queue and put it into a transaction table in a database. You can then massage and spread the data into an OLAP cube by processing this database table using stored procedures, for example.
Once the Gateway transactions are processed from a transaction table into the OLAP cube, reporting tooling can use the OLAP structures to display information such as:
  • charts and reports
  • trending information about how many requests have been processed by the Gateway
  • users who sent those requests
  • how many times the requests were processed or denied, and for which users.
Sample Audit Sink Policy
Once the audit sink policy has been enabled, you can edit it the same way that you would edit any other policy that is related to security processing in the Policy Manager. You have full access to all the same policy assertions you have when composing other Gateway policies.
The following is a sample audit sink policy:
Sample audit sink policy.png
The audit sink policy can access any context variable that was set during the execution of your Gateway's service policy, plus special Audit Sink Context Variables.
After your service policy has finished processing the incoming request, the Gateway runs the audit sink policy, which can access the context variables set in the main service policy. In the example policy shown above, information such as the routing latency and elapsed times were set in context variables during security policy execution. The Gateway then references and writes the information to the database in the  Perform JDBC Query Assertion, along with other information such as the user ID that was used to access the service.
Having the Gateway write information to a database is ideal when setting up an audit sink policy in a development environment. You can use a development database to capture the testing information and perform queries. Once the audit sink policy is migrated to a production environment, the information can be assembled into an XML message or custom message type. The Route via MQ Native Assertion to drop the information to a queue for processing.
Dropping messages into a queue in production is recommended, as it helps minimize outbound overhead from the Gateway by using a "fire and forget" method of passing on the audit information.