Examine Individual Components and Trace Data

You can examine detailed information about your trace data to find why your transactions are slow or broken. The Transaction Trace Viewer displays various representations of the data to help you identify components that are performing poorly.
You can examine detailed information about your trace data to find why your transactions are slow or broken. The Transaction Trace Viewer displays various representations of the data to help you identify components that are performing poorly.
Follow these steps:
  1. From the map or Analysis Notebook navigate to the
    Business Transactions
  2. Click the
    Business Transactions
  3. Click a row in the Transaction Trace list.
    The Transaction Trace Viewer displays four panes to show information about the transaction trace and components.
    • Transaction traces
      display in the upper pane.
    • The
      Transaction Trace Detail
      displays in the lower left pane.
    • The lower right pane is split into two sub-panes:
      • The
        Transaction Information
        about the selected transaction trace displays in the upper right sub-pane.
      • The
        Component Details
        display in the lower right sub-pane.
    Use the following graphic and corresponding legend to understand the various features in the Transaction Trace Viewer. The Transaction Trace Viewer contains four panes that show information about transaction traces and their components.
    The following legend identifies each item by number and provides more information:
    More Information
    Transaction Traces
    Examine each transaction trace. If there is more than one, information about the selected transaction trace displays in the other panes.
    Transaction Trace Detail pane
    This pane shows the graphical stack, which is the transaction order of the components from top to bottom. We can refer to the graphical stack as the "wedding cake" because the representation can look like an upside-down wedding cake.
    Transaction Information pane
    This pane shows information about the selected transaction trace. Notice that the path to the right of the drop-down list is the transaction trace path selected in the Transaction Trace pane.You can choose these views from the drop-down list:
    Browser Timing
    System Properties
    , and
    Component Details
    This pane provides information about the component that you selected in the graphical stack. When you have not selected a component, the information is about the top component in the graphical stack.
    The path located above the Transaction Trace Viewer.
    The graphical stack and transaction trace data show information to help you evaluate performance, for example:
    Calling relationships between components
    The rows for components are displayed from top to bottom in calling order.
    Call and component are synonyms. When you select a component, you are selecting a call in the transaction.
    A call can make child calls, which make the stack or depth:
    1 row = 1 or more calls = 1 or more components that can make child calls or invocations
    Tree view shows a one-to-one match of the data:
    1 row = 1 call = 1 component
    Summary view shows aggregated data, not a one-to-one match:
    1 row = path
    Path is the label for the call and the level in the stack. For example, a call "login" was made 5 times at the same stack level. You see 5 components in the same row. In Tree view, you see 5 rows. The Summary view count is 5, and the Total Duration column shows the sum of all 5 call times.
    Errors and Snapshots
    The red square containing white minus sign icon Red square with minus sign error icon indicates a component with an error.
    The camera icon Snapshot icon identifies a transaction with a related error or stall snapshot.
    For more information, see Analyze Errors and Stall Snapshots.
    Entry points
    Entry points typically appear as the first component in a transaction. If the transaction is a business transaction, the business transaction appears as the first component followed by the entry point.
    Transaction sequence
    The placement of components from left to right indicates sequence. Relative clock time in milliseconds appears across the top of the transaction. If the trace time is large enough, the time shows in seconds, hours, and days.
    Cross-process and cross-JVM transaction data
    A unique identifier, the correlation ID, links traced frontend and backend transactions. The sequence is based on the order in which frontends call backends in a transaction. The correlation ID helps you gain insight into which calls might be the source of a slow or stalled transaction. In the Transaction Trace Viewer
    Component Details
    , the correlation id is the Cross Process Data property value.
    Multiple graphical stacks in the viewer show related processes for the selected trace event. Processes that are traced from different agents appear in different shaded areas. Examine the linked components in a transaction trace to view cross-process or cross-JVM transactions. For example, see whether a certain transaction left one process and then the transaction entered a different process. You can also get insight into which calls could be the source of slow and stalled transactions.
    For more information about transaction correlation, see Use Cross-Process Transaction Tracing to Solve Problems.
    Deep visibility components
    A lightning bolt icon indicates a deep visibility component. This type of component is a method or component that is discovered automatically without the use of ProbeBuilder Directives. Application Performance Management analyzes methods for their complexity to determine the calls and components to instrument and display as deep visibility components.
    Run-time gap visibility
    With run-time gap visibility, agents automatically instrument and monitor applications based on their run-time performance.
    For more information, see Configure Smart Instrumentation.
  4. (Optional) Perform one or more actions:
    • Type a text string for filtering traces in the Search field. The search matches on information in all the transaction traces and components. The results display in all the views
      and any of the properties in the
      Component Details
      . Search also returns partial matches. For example, the search criteria is for
      . The
      view shows traces for
      , and
      Both the graphical stack and the views show all the related components. The components that match the search are highlighted in yellow in the stack. Where applicable in the views, matching text has a yellow background.
    • Mouse over the
      information icon
      (the blue dot with an i).
      A tooltip shows the fully qualified agent name. The information icon and corresponding row represent each individual agent.
  5. Examine the transaction traces from top to bottom to understand the traces that the agent captured.
    If an excessive number of traces were collected, you are informed that the Transaction Trace Viewer cannot render them. (For example, in Internet Explorer, a popup explains that the JSON file is too large to upload.)
    • Mouse over a trace or component to display tooltip shows the summary of the transaction trace information. When only a part of the component shows, the tooltip is useful.
    • Use the navigation functions to adjust your view.
      Click here for navigation help...
      Zoom In (Shift + Equals) changes the scale to show more detail.
      Zoom Out (Shift + Minus) changes the scale to show less detail.
      Reset (Shift + R) returns the viewer data to its original state.
      Center (Shift + C) centers the viewer data.
      Pan Left (Shift + Left) moves the focus to the left.
      Pan Right (Shift + Right) moves the focus to the right.
      Select Next (Down Arrow) selects next row.
      Select Previous (Up Arrow) selects previous row.
      Use the splitters separating the panes
      to display more of the pane of interest. Double-click a splitter to minimize or maximize the top and lower left panes.
      Click a
      Dock Size
      option to increase or decrease the size of the left and right panes relative to each other.
      Notice that a selection in any representation is automatically selected in the other representations when there is associated information, and conversely. For example, click a row in the graphical stack. The graphical stack, Transaction Information item,
      Component Details
      , and related views show information that match that row.
  6. Click a transaction trace of interest.
    You can select only one transaction trace at a time.
  7. Examine the graphical stack components from top to bottom to understand the call sequence within a period. An abnormally long component transaction time can indicate the problem root cause. Notice that the name and first Java source file line number of the instrumented Java method appears in parentheses.
    Red rows indicate an error condition. Light blue rows indicate that there is no problem. The light orange rows indicate that differential analysis triggered an alert. The darker orange rows indicate a stall.Yellow rows indicate a search result
    Transactions lasting 0 milliseconds display as a dot in a light blue background.
  8. Click a component of interest.
    In the right lower pane, the
    Transaction Information
    Component Detail
    panes display detailed information in their respective sub-panes.
  9. In the upper sub-pane, select an option from the
    drop-down list to show various representations of the data.The selection behavior varies for each view. For example, the
    view lists transaction trace paths, while the
    Browser Timing
    shows data for hard pages, and resources associated with soft pages. Therefore the information you can select for these views is different.
    view lists trace paths in hierarchical call order. Each path includes the names of the classes and methods of a call. You can view the changes that occurred between a starting point and an ending point of a trace. Path, Timestamp, Duration, Percent, and Invocations data appear in sortable columns. (Source Line is not sortable.) Some traces for a component might not have source line information. In this case, N/A (Not Applicable) shows in the Source Line column. Adjust how much information appears by expanding or collapsing paths. Colored icons indicate the trace type. For example, light orange Differential Analysis light orange indicates that Differential Analysis triggered an alert--a transaction has uncontrolled variance. The
    view lets you step through your data examination and navigate to specific components to identify performance problems. Data values help you understand how an initial value is affected by a series of intermediate values.
    view displays data in which colors indicate the trace type. (The higher call ratio values are deep shades because calls render on top of each other.) This visual summary lets you quickly identify problem areas at a glance. After you identify a problem, you can use the
    view to investigate the details of that type of call.
    The column data shows totals for each row. For example: A component is called 100 times at the level 4 row of a transaction. The level 4 row-count is 100. Each call is 2 milliseconds, so the total duration is 200 milliseconds (100 x 2). The total duration value in the breadcrumb can differ from the total duration value in the
    . This difference can occur because the breadcrumb value includes all correlated traces from a backend and the
    view value does not.
    Browser Timing
    Browser Timing shows data for hard pages and their resources and for resources that are associated with soft pages. (The data for a soft page itself does not appear.)
    For more information about Browser Timing, see Diagnose Resource Load Issues.
    System Properties
    System Properties provides CPU timing and thread contention monitoring data for transaction traces in the servlet method.
    For more information about System Properties, see Diagnose System Performance Issues.
    Snapshot provides information about error stall snapshots.
    For more information about error and stall snapshots, see Analyze Error and Stall Snapshots.
  10. Examine the
    Component Details
    Click here for component details...
    Not all the information about deep visibility components is available.
    --The name of the component, for example: Default
    --The full resource name of component, for example:
    --The execution time in milliseconds of the selected component
    (relative)--The start time (based on the system clock) of the invocation of the root component
    Duration Percent
    --The percentage of total transaction time
    System Properties
    --Timing data lets you diagnose method threads for performance issues. Colored boxes identify the various system properties. A bar chart represents the data values for these system properties. When the values are 0 ms, the chart shows a black line. Each system property and its value are listed below the chart.
    --A list of the component optional properties, including the following property:
    • Source Line-- Java Agent Only -- The name and first Java source file line number of the instrumented Java method. The Java Agent does not collect the line numbers of code that call the instrumented Java method. Standard exception thread stack dumps in debuggers and core dumps typically display these line numbers.
    to expand the entire tree. Click
    to collapse the entire tree. Click the right-facing arrow next to a component to show a single sub-component.
    Resize any column to show more or less information.
    Mouseover a
    column name
    or a
    system properties chart segment
    to see a tooltip with information about the item.
  11. Determine the root cause of the application performance problem. Use the identifying information to gather specific details about the problem. In general, you are looking for the deepest, longest running components in your trace. For example:
    • A single call to a database
    • A flurry of quick calls that together add up to high latency
You can contact operations staff to request a review of the application code. The staff determines if a code issue or an application dependency issue exists.