Utilize Transaction Tracing

The transaction tracing function enables administrators and support engineers to trace any HTTP request coming into the CA API Developer Portal (API Portal) as a single transaction across its multiple services.
The transaction tracing function enables administrators and support engineers to trace any HTTP request coming into the CA API Developer Portal (API Portal) as a single transaction across its multiple services.
This function enables you to:
  • Help diagnose where and when a request failure occurs when debugging issues
  • Provide insight into service-to-service interactions
  • Identify service bottlenecks in performance evaluation and optimization
Transaction Tracing: How It Works
You can trace transactions coming into the following API Portal services:
  • Dispatcher
  • PSSG
  • Ingress
  • portal-data
  • Portal Enterprise
  • RBAC
  • Tenant Provisioner (TPS)
  • Authenticator
For more information about API Portal services, see API Portal Architecture.
The following HTTP headers are involved in a transaction tracing. You can view and search these headers within the log files (for on-premise deployments) or with tools such as Kibana (for SaaS deployments):
  •  
    TraceId
    The TraceId is encoded as 32 or 16 lower-hex characters and indicates the overall ID of the trace. Every request in a transaction shares this ID across multiple services.
    For example: 67842807e96db762b3211988ed426318.
  •  
    SpanId
    The SpanId is encoded as 16 lower-hex characters and indicates the position of the current operation in the trace tree. Every request in a transaction has a unique ID, or derives the ID from the TraceId if one cannot be generated.
    For example: 521c7d21629846d9806020308e25274c.
System logs for Portal services use the following standard format:
[Appender] [Date Time] [Log Level] [Abbreviated Logger Class] [Thread ID] [TraceID|SpanID] [TenantID|UserID|UserAgent|TxID] Log Message
System logs for Gateway services, which include PSSG and Ingress, use the following logging formats:
  • Gateway prefix line 
    [Date-Time]-[G/W Log Level]-[Thread ID]-[Logger Class]-Audit Code-?: Trace ID
  • Portal-log line
    [Gateway Component] [Date Time] [Log Level] [] [] [TraceID|SpanID] [TenantID|UserID|UserAgent|TxID] Log Message
With the exception of the Gateway services (PSSG and Ingress), all Portal Services define a set of logging appenders that isolate logs based on the tenant and purpose. The appenders are described in the following table:
Appender
Naming Convention
Appender Description
Tenant System Appender
${tenantId}-portal.log
Handles general system logs attributed to a specific tenant.
Tenant Audit Appender
${tenantId}-portal-audit.log
Logs Portal's audit events.
Tenant Performance Appender
${tenantId}-portal-perf.log
Logs performance details for a specific tenant.
System Appender
system-portal.log
Logs system-level events.
3rd Party Appender
portal.log
Defines a default appender for third-party library logging.
Examples
The following are sample log entries from the services to show an unsuccessful login transaction from 
apim
 to 
authenticator
, and then back to 
apim
, with an ERROR occurring during authentication. 
Note the shared TraceId (in red), the SpanId headers (in blue) logged as the request goes through downstream services, and the time stamps tracking the transaction.
apim:
[ingress] [2018-09-14T17:11:30.696+0000] [INFO] [] [] [67842807e96db762b3211988ed426318|521c7d21629846d9806020308e25274c] [||] custom-message-received via port 80, transaction entry
[ingress] [2018-09-14T17:11:30.699+0000] [INFO] [] [] [67842807e96db762b3211988ed426318|521c7d21629846d9806020308e25274c] [||] Policy Fragment: portal-message-received
authenticator:
[system-portal.log] [2018-09-14 17:11:30,961] [INFO ] [c.c.a.s.a.i.PdbImpl] [qtp211090736-816] [67842807e96db762b3211988ed426318|140d540e8a540031] [||Layer7-SecureSpan-Gateway/v9.3.00-b8670_CR03|] Successfully fetched the user details associated with username: admin
[system-portal.log] [2018-09-14 17:11:30,983] [WARN ] [c.c.a.s.a.i.PdbImpl] [qtp211090736-816] [67842807e96db762b3211988ed426318|140d540e8a540031] [||Layer7-SecureSpan-Gateway/v9.3.00-b8670_CR03|] The password entered by user: admin does not match. Incrementing the invalid login count
[system-portal.log] [2018-09-14 17:11:30,992] [INFO ] [c.c.a.c.i.AuthControllerImplV2] [qtp211090736-816] [67842807e96db762b3211988ed426318|140d540e8a540031] [||Layer7-SecureSpan-Gateway/v9.3.00-b8670_CR03|] Total time elapsed in AuthenticateV2 operation [97 ms (~0 secs)].
[system-portal.log] [2018-09-14 17:11:31,047] [INFO ] [c.c.a.a.ApplicationTransactionInterceptor] [qtp211090736-816] [67842807e96db762b3211988ed426318|140d540e8a540031] [||Layer7-SecureSpan-Gateway/v9.3.00-b8670_CR03|] transaction exit
apim:
[ingress] [2018-09-14T17:11:31.053+0000] [INFO] [] [] [67842807e96db762b3211988ed426318|521c7d21629846d9806020308e25274c] [||] [ingress, message-completed] request.url = http://tenant1.example.ca.com/tenant1/authenticate/login, request.http.method = POST, remoteHost = 10.128.16.1, request.time.millis = 1536945090691, elapsedTimeMs = 351, request.size = 81, response.size = 131, response.http.status = 401, routingStatus = None, transaction exit
[ingress] [2018-09-14T17:11:31.050+0000] [ERROR] [] [] [67842807e96db762b3211988ed426318|521c7d21629846d9806020308e25274c] [tenant1||] Authenticator Login Service: authentication failed
[ingress] [2018-09-14T17:11:30.702+0000] [WARN] [] [] [67842807e96db762b3211988ed426318|521c7d21629846d9806020308e25274c] [tenant1||] authenticator login service, tenant_id = tenant1
The following are sample log entries for PSSG and Ingress:
  • Gateway prefix line
    [2018/09/05-18:26:18,027]-[INFO ]-[284]-[com.l7tech.log.custom.tenantLogger]--4: 5a58b96aeb8aaf56e2c7e7daaa66a062
  • Portal-log line
    [ingress] [2018-09-05T18:26:17.786+0000] [WARN] [] [] [5a58b96aeb8aaf56e2c7e7daaa66a062|7e69b4b0663646e09ed8c669447579c2] [apim||] authenticator login service, tenant_id = apim
    [pssg] [2018-09-05T18:21:00.216+0000] [WARN] [] [] [5a58b96aeb8aaf56e2c7e7daaa66a062|7e69b4b0663646e09ed8c669447579c2] [apim||] trace-reference