Collect and Analyze Transaction Traces

As a diagnoser, triager, or administrator, you view detailed information about transaction trace components such as methods. This information helps you to identify the root cause of application performance problems. With the Transaction Tracer, you can examine automatically or manually collected transaction traces. Information appears because the components are instrumented using PBDs, or smart instrumentation is enabled.
apmdevops106
3
This diagram shows how to identify the root cause of application performance problems by viewing detailed information about transaction components.
How to collect and analyze transaction traces
How to collect and analyze transaction traces flow diagram
Transaction Tracing Overview
The Transaction Tracer monitors the activity of individual transactions as they flow through the boundaries of supported agents. Agents support monitoring of Java, .NET, and Node.js applications. Cross-process transaction tracing monitors transaction calls between different Java Virtual Machine (JVM), Common Language Runtime (CLR), and Node.js instances. The monitoring can occur on local or remote computers. Viewing cross-process transaction details helps you carry out problem investigation across transaction processes.
Here are the ways to collect transaction traces:
After a transaction trace session starts, the transactions that match filter criteria appear in Workstation or WebView. You can view the trace information in the Transaction Tracer tab, including transaction events such as transaction traces and errors.
More information:
See KD10688:Ways to Contain/Control Transaction Traces to learn how to handle excessive transaction traces.
Automatically Collected Transaction Traces
When smart instrumentation is enabled, Introscope automatically collects a transaction trace when these actions take place:
  • Differential Analysis detects application instability
    Note:
    Automatic transaction traces from differential analysis require agent version 10.0 or greater.
  • An error occurs
  • The
    ComponentTimeAutoTraceTriggerTracer
    tracer is deployed and a component response time is exceeded
  • An API triggers an automatic transaction trace that is based on custom criteria.
    More information:
    Contact CA APM Implementation Services.
Highly optimized, low-overhead tracers collect automatic transaction traces. Automatic traces incur much less performance overhead than manually run, sample, and other transaction traces. These other types of transaction traces use agent filtering that adds overhead.
Automatic transaction traces have these characteristics:
  • Are supported for the Java, PHP, and Node.js agents. The .NET agent is not supported.
  • Every PBD-instrumented component has an associated metric in the Investigator tree. Deep visibility components do not have an associated metric.
  • Deep visibility components contain only class name, method name, and duration. For frontends and backends (for example, servlets, web services, SQL calls), the name is formatted based on the PBD configuration.
  • Include cross-process transaction traces
    Note:
    Cross-process transactions in automatic transaction traces are supported only for Java applications.
  • Automatic transaction traces do not display all of the component detail properties that the other types of transaction traces display.
  • The following properties display in the Component Properties:
    • The Trace Type is Normal.
    • If an error triggered the transaction trace, the component that threw the exception includes the component properties.
    • The first component in the transaction trace displays the
      Auto Trace Trigger Criteria
      property.
If you do not see automatic transaction traces, it could be for one of these reasons:
  • Smart instrumentation is not enabled.
  • A legacy mode agent is monitoring the JVM. Legacy mode does not support smart instrumentation.
  • Number of automatic transaction traces per interval (
    agent.deep.automatic.trace.clamp property
    ) clamp value is exceeded.
    When manually running a trace and any manual filters match an automatic trace, Introscope collects only the manual trace.
Cross-Process Transactions in Automatic Transaction Traces
Viewing cross-process transactions in automatic transaction traces helps you evaluate cross-tier transactions. You can determine the tier that is the bottleneck for a problematic transaction.
Cross-process transactions in automatic transaction traces are supported only for Java applications.
Upstream and Downstream Agents
Agents that monitor transactions can be upstream, downstream, or both upstream and downstream from other agents monitoring the same transaction. Upstream agents can make cross-process calls to downstream agents. Transaction calls can traverse more than two processes. Therefore, agents monitoring processes between the first and last transaction calls are positioned both upstream and downstream from other agents. Here is an example:
  1. You are reviewing a transaction trace for a problematic transaction and notice a 6-second (6000 ms) execution time.
  2. In the
    Trace View
    , you see calls from the
    dataservice.yourcompany.net/invoke
    client-side method going to to the
    thirdparty.mycompany.net/invoke
    server-side method.
  3. You notice that the server-side method is making a large number of calls to the
    3PP site data
    third-party web service, which is instrumented.
  4. The
    Trace View
    shows that the
    3PP site data
    web service backend is servicing repeated requests in rapid succession. This behavior indicates that programming logic, such as a nested loop, is likely causing repeated calls in the server-side service. You determine that the server-side invoke operation is responsible for most of the overall transaction-execution time.
This graphic shows the agent positions in the example cross-process transaction.
Diagram of Upstream and Downstream Agents
Diagram Showing Upstream and Downstream Agents
Agents report problems when they encounter them, and sometimes agents start transaction traces. Before automatic transaction tracing included cross-process calls, only upstream agents could report problems and could collect transaction traces.
Visibility Before Cross-Process Tracing in Automatic Transaction Traces
An agent near a transaction beginning does not know if the transaction has a downstream problem.  At the transaction finish, a downstream agent reports any downstream problem to the closest upstream agent. All that upstream agents know is that the transaction is unfinished until a downstream agent reports the finish. At transaction completion, the upstream agent nearest the transaction beginning reports found problems to the Enterprise Manager as transaction traces. Before automatic transaction tracing included cross-process call monitoring, downstream agents detecting problems never sent transaction traces to the Enterprise Manager. If problems discovered by downstream agents could not get reported up to the topmost upstream agent, those problems went unreported. This situation resulted in a significant lack of visibility into application problems.
End-to-end Visibility with Cross-Process Tracing in Automatic Transaction Traces
With cross-process tracing supported, Java agents can use intelligent caching to send automatic transaction traces for problematic downstream cross-process transactions. This capability provides end-to-end visibility for Java transactions.
Both upstream and downstream agents that detect problematic transactions can trigger automatic transaction traces. They can both also send transaction trace information to the Enterprise Manager. Agents that trigger automatic traces decide at the end of transaction whether to send a trace to the Enterprise Manager. This decision timing is especially useful for differential analysis, when Enterprise Manager baseline thresholds trigger automatic transaction traces.
Transactions can use a supported protocol, for example, HTTP or SOAP, to invoke a service within the same process. In this situation, the agents making cross-process calls can be positioned as both upstream and downstream from other agents. When an upstream agent collects an automatic transaction trace, all participating downstream agents also collect automatic transaction traces. Cross-process transaction tracing collects only the traces important to you for triaging problems for root cause analysis.
When an upstream agent triggers a trace, the
Component Details Auto Tracing Triggered
property displays the trigger type. For example, an error. When a downstream agent triggers collection of automatic traces, the
Component Details Auto Tracing Triggered
property value is
Cross Process Trigger.
Example: Agent Cross-Process Transaction Trace Correlation in Automatic Transaction Traces
An agent that is upstream from others agents detects a problem. The upstream agent triggers collection of an automatic transaction trace, and notifies downstream agents to do the same. All the agents send their automatic transaction traces to the Enterprise Manager. Together, the agents create a correlated cross-process transaction, as shown in the following diagram.
Automatically Triggered Correlated Cross-process Transactions Traces
Diagram Showing Automatically-Triggered Correlated Cross-process Transactions Traces
Automatic transaction traces also support cross-thread correlation within the same process. Thread calls can traverse more than one transaction. One transaction in a process can go through different threads. In the following graphic, you can see four cross-process threads in the bottom row of an automatic transaction trace.
Threads in an Automatic Tranaction Trace Cross-process Transaction Image 1.png
Upstream threads can trigger automatic transaction traces anytime, either before or after the downstream threads are invoked. The transaction details depend on whether the thread is upstream or downstream of the initial triggering thread in the transaction.
Example: Thread Cross-Process Transaction Trace Correlation in Automatic Transaction Traces
An upstream thread detects a problem. The upstream thread triggers collection of an automatic transaction trace, and notifies downstream threads to do the same. All the threads send their automatic transaction traces to the Enterprise Manager. Together, the threads create a correlated cross-process transaction.
Cross-process transaction in auto trace 2
Threads in an Automatic Tranaction Trace Cross-process Transaction Image 2
Automatically Collect a Transaction Trace Based on Application Instability
Differential Analysis tracks the stability of your applications. By default differential analysis looks for uncontrolled variance in the Average Response Time of your frontend and business transaction metrics. The Differential Analysis map is a visual exploration of the stability and responsiveness of many applications. Each strip in the map corresponds to a single metric. In a quiet, stable period the strip is a light shade. When stability degrades, the shade of the strip darkens progressively according to the severity of the instability. Thus, a single strip lets you see the stability of a single application or business transaction over time. The map sorts the most unstable strips to the top.
When an application becomes slightly unstable differential analysis notifies the agent, which is prepared to collect related automatic transaction traces. Typically the agent collects transaction traces until the application becomes stable. However, the agent can decide not to collect a transaction trace when the instability period is brief. View these transaction traces for detailed information specific to the stability change and to gain insight into the root cause.
Follow these steps:
  1. In APM Team Center, click
    WebView
    .
    APM WebView appears.
  2. Click
    Investigator
    ,
    Metric Browser
    .
    A tree shows a hierarchical view of your system.
  3. In the tree, select the agent for which you want performance information, for example:
    SuperDomain | Host | Process | Agent | Frontends | Apps | App Name
  4. Click the
    Differential Analysis
    tab.
    The map shows a graphical representation of performance data and shows the last 8 minutes of data. The top 100 problematic metrics appear in decreasing instability. The data refreshes when you query, change the time period, or select a different node.
  5. Click a strip of interest in the map.
    The Differential Analysis chart appears. This chart helps you understand the metric stability over the timeline that the strip represents. The chart shows the status of the monitored component, so you can quickly detect normal and abnormal performance:
    A line represents an actual metric value.
    The shaded regions correspond to standard deviation bands 1, 2, and 3. The darker the band, the higher the deviation from the predicted value. Any metric in the white is better than predicted. Any metric that appears above the bottom white area has exceeded the predicted value. For example, if the metric exceeds the top band, it is exceeding 3x the standard deviation.
  6. Mouseover over the line.
    Tooltips display the metric values.
  7. Click the hyperlink for the metric of interest, for example, Average Response Time.
  8. In the metric browser, click the folder node directly above the metric, then click the
    Traces
    tab.
    You can view the automatic transaction traces generated by the instability.
  9. (Optional) Click other nodes to see transaction traces that Differential Analysis generated.
    For the example, click the
    Frontends
    node, then the
    Traces
    tab to see automatic traces for all your frontend applications.
You can configure differential analysis for entry points and other application metrics.
Trigger Automatic Transaction Traces When Component Response Time Exceeded
You can deploy a PBD to trigger a transaction trace automatically when the component response time has been exceeded. You create a PBD entry with a tracer to collect this automatic trace. For example, to collect a trace when a specific servlet response time exceeds 10 seconds. Use the
ComponentTimeAutoTraceTriggerTracer
tracer option to configure this capability.
Manually Collect a Transaction Trace
To run a transaction trace session manually, specify the agents whose transactions you want to trace, and the data-capture period. Specify filters to limit tracing to transactions that have exceeded the threshold execution time, matched parameter values, or contain errors. After the transaction trace session starts, the transactions that match the filter criteria appear in the transaction table. Transaction events include transaction traces and errors.
You must have
run_tracer
permission to run a new transaction trace session.
Follow these steps:
  1. In APM Team Center, click
    WebView
    .
    APM WebView appears.
  2. Click
    Tools
    ,
    Transaction Tracer
    .
  3. Click
    Start Trace Session
    .
  4. In the Trace Transactions area:
    • Specify the minimum duration for the transaction trace. Select milliseconds or seconds from the drop-down list. The default is 5000 milliseconds.
    • (Optional) Specify one of the following filter conditions for the transaction trace:
      • Equals
        The parameter value that matches the strings that are traced.
      • Does not equal
        The parameter value that does not match the specified string are traced. Transactions that do not include the parameter to which the filter applies are also traced.
      • Contains
        The parameter value that contains the specified string are traced.
      • Starts with
        The parameter value that starts with the specified string are traced.
      • Ends with
        The parameter value that ends with the specified string are traced.
      • Exists
        Transactions that include the parameter to which the filter applies are traced, regardless of the parameter value.
      • Does not exist
        Transactions that do not include the parameter to which the filter applies are traced.
  5. Enter the trace session duration.
  6. In the
    Trace Agents
    area, select one or more agents for which to trace transactions:
    • To trace all agents, click
      Trace all supported Agents
      . This option traces supported agents that are currently connected, and any that connect during the Trace session.
    • To trace selected agents, click
      Trace selected Agents
      and select agents from the list (
      CTRL + click
      to select multiple agents).
  7. Click
    OK
    to start the transaction trace session.
    After the session starts, the status bar in the lower panel displays the following information:
    • Number of traced transactions and a brief description of the trace configuration.
    • Time remaining in the session.
    In Live mode, the
    Traces
    tab lists transaction trace events from the last 20 minutes. Transaction trace event older than 20 minutes are not displayed in live mode. No more than 500 transaction trace events are listed.
Stop, Restart, or Switch Transaction Trace Sessions
You can manage the trace session from the
Transaction Tracer
tab as follows:
  • Click
    Stop Trace
    to end the session.
  • Click
    Restart Trace
    to continue to trace transactions in the targeted agents using the same conditions. You can restart a transaction trace session:
    • After a session has timed out.
    • To restart a session that you have stopped.
    • To restart an in-progress session.
  • Click
    Switch Trace
    to select a different trace session to run.
Find the Relevant Trace
You examine a single trace to gather information about the application problem.
Follow these steps:
  1. Analyze the trace activity information for the trace session in the transaction table. Select a table row to view more details.
  2. Run new or stop, restart, or switch trace sessions as needed.
  3. Examine the
    Summary View
    ,
    Trace View
    , and
    Tree View
    tabs in the lower pane. The information can help you to diagnose and triage application problems and performance issues.
Understand Entry Points
Automatic entry point discovery allows you quickly to monitor and triage Java applications without manually configuring ProbeBuilder Directives.
When smart instrumentation and entry point detection are enabled, Introscope monitors threads that are involved in client socket call transactions.  Smart instrumentation and entry point detection are configured as enabled by default.
Entry points
are the transaction beginning points. A rules engine within the agent identifies entry point candidates. The agent instruments and monitors the earliest entry point candidate in the transaction thread. Any entry point that an agent detects and persists is enabled for monitoring by all agents sharing the installation directory. However when multiple agents share instrumentation, metrics reporting depends on application server JVMs executing the same code or framework classes.
Here are examples of transactions into which entry point detection automatically provides visibility:
  • Technology stacks and frameworks that Introscope instrumentation does not already monitor
  • Custom or proprietary API calls
  • Background threads that consume critical resources and can affect the overall application performance
: Entry point detection does not support User Datagram Protocol (UDP).
The agent saves entry points in the
AutoPersist.pbd
, which is persisted in the <
Agent_Home
>\core\config\hotdeploy directory.
 The system user running the application server requires read/write access to the \hotdeploy directory. These permissions allow the agent to write data to
AutoPersist.pbd
.
Entry points differ from frontends. The Java agent automatically discovers entry points, which are near the start of a transaction call path. Frontends are manually defined in PBDs, and can be anywhere in a transaction call path.
 Do not make manual changes to the
AutoPersist.pbd
file. However, you can copy discovered entry points and can use them in a different PBD.
Entry points metrics display in the agent-centric tree in the Automatic Entry Points subnode under the agent node.
Ensure the
introscope.autoprobe.dynamicinstrument.enabled
property in the
IntroscopeAgent.profile
is set to
true
. This configuration allows the agent to report entry point metrics without requiring an application restart.
Introscope reports the five standard blame metrics for each entry point. Entry points display in transaction traces, but do not display on the Map. Entry point names are formatted as
entry point <
class name _ method name
>
. Introscope reports entry point supportability metrics.
You can configure entry points collection. For example, a configuration property limits the number of entry points that
AutoPersist.pbd
can persist.
More Information
:
Create New PBDs to Convert Entry Points to Frontends.
Understand Automatic Backends
When
automatic backend
discovery is enabled, the agent automatically discovers and monitors application backends without manual configuration.
A discovery engine within the agent identifies automatic backend candidates. Any automatic backend that an agent detects and persists is enabled for monitoring by all agents sharing the installation directory.
Here are examples of backend types that automatic backend detection can find and monitor:
  • Backend technology stacks and frameworks that the agent does not already monitor. Examples are NoSQL backends such as MongoDB and Cassandra.
  • Custom or proprietary backends
The agent saves automatic backends in the
AutoPersist.pbd
, which is persisted in the <
Agent_Home
>\core\config\hotdeploy directory.
  • The system user running the application server requires read/write access to the /hotdeploy directory. These permissions allow the agent to write data to AutoPersist.pbd.
  • Do not make manual changes to AutoPersist.pbd. However, you can copy discovered automatic backends and can use them in a different PBD
You can configure automatic backend detection. For example, a configuration property limits the number of automatic backends that AutoPersist.pbd can persist.
In the
Trace View
, the r
emote port
and
remote server name
parameters display in the
Component Details
.
Notice the backend call path and the parameters in this transaction trace:
  • The transaction trace component for the backend call path is: Backends|com.mysql.jdbc|MysqlIO|sendCommand.
  • The remote port parameter is
    3306
    and the remote server host parameter is
    Name1: localhost
    .
Automatic Backend Displayed in a Transaction Trace.
Understand Deep Visibility Components
When smart instrumentation is enabled, agents automatically discover and collect detailed information about transaction components to the method level. Agents discover and automatically instrument deep visibility components without the use of ProbeBuilder Directives (PBDs). Introscope analyzes methods for their complexity to determine the calls and components to instrument and display as deep visibility components.
Note:
Smart instrumentation is available only for Java agents, not .NET agents.
Things to know about deep visibility components:
  • A lightning bolt icon indicates a deep visibility component. The Deep Trace Component label also appears in the tooltip when you mouse over such a component.
  • Deep visibility components do not include links to metrics. No metric data displays in the Investigator tree or on the Map.
  • Contain only class name, method name, and duration.
  • The following properties display In the component details:
    • Instrumentation Level
      Smart instrumentation level at which a transaction was discovered.
    • Method Level Score
      Smart instrumentation level that correlates to the score that the Introscope scoring algorithm assigns to a deep visibility component method. Introscope can display deep visibility component methods having varying scores within a transaction trace, error, or stall. For example, a transaction discovered using medium level can display methods with method level scores of both medium and low.
Use these properties to understand the smart instrumentation visibility depth of a transaction trace and methods within a trace. For example, you can compare the number of methods that smart instrumentation discovers at two different instrumentation levels. You can note the smart instrumentation level at which Introscope scored specific methods. You can adjust the monitoring solution for the desired monitoring visibility balancing overhead and visibility.
Both standard instrumented components and deep visibility components can appear through a transaction trace session, or when a transaction trace is started using the Command-Line Workstation.
Depending on your requirements and environment, you can configure the depth of smart instrumentation visibility.