Create Custom Tracers

You can further refine your metric collection by creating custom PBD files. Creating custom directives, by creating tracers to track application-specific measurements, require the use of specific syntax and keywords. 
apmdevops106
You can further refine your metric collection by creating custom PBD files. Creating custom directives, by creating tracers to track application-specific measurements, require the use of specific syntax and keywords. 
2
2
To write custom tracers, you must define:  
  • The directive type (indicating generically how many classes or methods to trace)
  • The specific classes or methods to trace
  • The type of information to trace in one or more classes or methods (for example, a time, a rate, or a count)
  • The fully qualified metric name (including the resource path) under which to present this information
Custom PBDs are stored in the
<Agent_Home>
/core/config/hotdeploy directory. Any PBDs added to this directory is implemented without having to update or modify the
introscope.autoprobe.directivesFile
property in the IntroscopeAgent.profile. If you have enabled dynamic ProbeBuilding, the PBDs in the hotdeploy directory are picked up live from the folder. No reboot is required.
Once a custom PBD is created, Introscope treats it as if it is an out-of-the-box PBD. You can set alerts on the metrics that are created and can use the metrics to create custom dashboards.
Select
methods to trace carefully, as more methods traced means more overhead.
Using a custom BlamePoint Tracer tracer for common metrics
A
BlamePoint Tracer
is the most commonly used tracer. This tracer generates five separate metrics for associated methods or classes:
  • Average Response Time (ms)
  • Concurrent Invocations
  • Errors Per Interval
  • Responses Per Interval
  • Stall Count
Here is an example of a BlamePoint Tracer. A BlamePoint Tracer is set for a method that is named 
searc
h
in the 
petshop.catalog.Catalog
class. The BlamePoint metrics display under the PetShop|Catalog|search node in metrics browser tree.
TraceOneMethodOfClass: petshop.catalog.Catalog search BlamePointTracer "PetShop|Catalog|search"
Directive Names and Arguments that are used in Tracer Syntax
PBD files contain simple keywords that associate tracers into groups or enable and disable groups. PBD files also contain tracer definitions. Custom tracers require specific syntax, which allows Introscope to recognize and process the tracers.
A tracer is composed of a directive and information about the method or class to trace, in the following format:
<directive>: [arguments]
Where
[arguments]
is a list and directive-specific.
Only a subset of the parameters is required depending on the directive that you use.
The most common directives to use are the following trace directives:
  • TraceOneMethodOfClass
    This directive traces a specified method in the specified class.
  • TraceAllMethodsOfClass
    This directive traces all methods in the specified class.
  • TraceOneMethodIfInherits
    This directive traces a specific method in these two situations:
    • In all direct sub-classes of the specified class 
    • In all direct interface implementations of the specified interface. 
    The specified class or interface should be the fully qualified name.
  • TraceAllMethodsIfInherits
    This directive traces all methods except for these components:
    • Constructors and static initializers in all direct sub-classes of the specified class 
    • Direct implementations of the specified interface. 
    The specified class or interface should be the fully qualified name.
  • TraceOneMethodIfFlagged
    Use this property when the specified class is included in a tracer group that has been enabled with the
    TurnOn
    keyword. This directive traces one method when the conditions in the previous sentence are met.
  • TraceAllMethodsIfFlagged
    Use this property when the specified class is included in a tracer group that has been enabled with the 
    TurnOn
     keyword. This directive traces all methods except for constructors and static initializers when the conditions in the previous sentence are met.
  • TraceComplexMethodsIfFlagged
    This directive traces all public or package-visible non-synthetic methods that call any other method, except constructors and static initializers in the tracer group.
  • TraceAllComplexMethodsIfFlagged
    This directive traces all methods, except constructors and static initializers, which call any other method, in the tracer group.
The
TraceAll*
directives instruments every method of the specified classes. Using these directives can lead to unnecessary overhead in the application and can result in transaction traces that exceed the default 5,000 component limit.
The
TraceAllComplex*
and
TraceComplex*
directives instruments classes that you identify in the directive or the associated tracer group.
The
TraceOne*
directives only apply instrumentation to the methods that you specify, not any of the other methods that it calls.
Only concrete, implemented methods can be traced and can report metric data while running. An abstract method that is specified in a custom tracer results in no metric data being reported.
The expected syntax for trace directives usually consists of the following arguments:
  • <Tracer-Group>
    The group to which the tracer is associated.
  • <class>
    A fully qualified class or interface name to trace. Fully qualified classes include the full assembly name of the class and the name, for example:
    [MyAssembly]com.mycompany.myassembly.MyClass
    The assembly name must be enclosed in [] brackets.
  • <method>
    The method name (for example,
    MyMethod
    )
    OR
    The full method signature with return type and parameters.
    For example,
    myMethod;[mscorlib]System.Void([mscorlib]
    System.Int32
    . For more information about method signatures, see Signature differentiation.
  • <Tracer-name>
    This directive specifies the tracer type to be used. For example, BlamePoint Tracer.
    More information: commonly used tracer names.
  • <metric-name>
    Controls how the collected data is displayed in the Introscope Workstation.
    The following examples describe ways to specify the name and location of a metric at different levels of the metrics tree.
    metric-name
    -- the metric appears immediately inside the agent node.
    resource:metric-name
    -- the metric appears inside one resource (folder) below the agent node.
    resource|sub-resource|sub-sub-resource:metric-name
    -- the metric appears more than one resource (folder) level deep below the agent node. Use pipe characters (|) to separate the resources.
Using Quotes in Custom Tracer Definitions
Custom tracers can contain metric names with spaces in them. When using spaces in your custom metric names, We recommend putting quotes ("") around all metric names.
Do not place quotes around class names. Quotes cause custom tracers to malfunction.
 Examples of Quotes Causing Custom Tracers to Malfunction
Correct
IdentifyClassAs: MyClass MyTracers
Incorrect
IdentifyClassAs: "MyClass" MyTracers
If you create a metric name that contains a class name, you must use quotes around the whole metric name. Metric names are allowed to have spaces, and all spaces in metric names must be contained within quotes. For example, represent the metric name "{classname}|Test One Node"  as follows:
Correct
TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer 
"
{classname}|Test One Node
"
Incorrect
TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer {classname}|Test One Node
Introscope does not monitor classes having invalid class file names.
Examples of Classes Having Invalid Class File Names
In the class file name:
org/jboss/seam/example/seambay/AuctionImage$JaxbAccessorM_getData_setData_[B:
The
_[B:
causes the class file name to be invalid. You cannot use an open square bracket
[
as part of the Java class file name. When Introscope encounters such classes having invalid class names, it fails to instrument them. Introscope also reports the invalid class names as an error message in the agent logs.
The following sections are examples of method tracers. In the following example, quotes ("") are used around the metric names. We highly recommend that you put quotes around all metric names when you create custom metric names.
Commonly Used Tracer Names and Examples
The following list describes the most commonly used tracer names and what they trace.
  • BlamePointTracer
    This tracer name provides a standard set of metrics for a blamed component including: average response time, per interval counts, concurrency, stalls, and errors.
  • ConcurrentInvocationCounter
    This tracer name reports the number of times a method has started but not yet finished. The result is reported under the metric name that is specified in the tracer,
    <metric-name>
    , in the Investigator tree. An example use of this tracer would be counting the number of simultaneous database queries.
  • DumpStackTraceTracer
    This tracer name dumps a stack trace to the instrumented application standard error for methods to which it is applied. The exception stack trace that is thrown by the Dump Stack Tracer is not a true exception. The exception stack trace is a mechanism for printing the method stack trace.
    This feature is useful for determining call paths to a method.
    This feature imposes heavy system overhead. We recommend that you use this tracer only in a diagnostic context where a sharp increase in overhead is acceptable.
  • MethodCPUTimer
    Average CPU time (in milliseconds) used during method execution and reports it under
    <metricname>
    in the metrics tree.
    Note:
    This tracer requires a platform monitor on the supported platform.
  • MethodTimer
    Average method execution time in milliseconds and reports it under the metric name that is specified in the tracer,
    <metric-name>
    ,
    in the metrics tree.
  • PerIntervalCounter
    Number of invocations per interval. This interval changes based on the view period of the consumer of the data (for example, the View pane in the Investigator). PerIntervalCounter is reported in the metric browser tree under the metric name that is specified in the tracer,
    <metric-name>
    .
Average Tracer Example
This tracer tracks the average execution time of the given method in milliseconds.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search BlameMethodTimerHC "Petstore|Catalog|search:Average Method Invocation Time (ms)"
Rate Tracer Example
This tracer counts the number of times the method is called per second, and reports this rate under the specified metric name.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search BlamedMethodRateTracer "Petstore|Catalog|search:Method Invocations Per Second"
Per Interval Counter Tracer Example
This method tracer counts the number of times the method is called per interval, and reports the per interval count under the specified metric name.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search PerIntervalCounter "Petstore|Catalog|search:Method Invocations Per Interval"
The interval is determined by the monitoring logic in the Enterprise Manager, such as the Graph frequency.
The preview pane in the Investigator defaults to 15-second intervals.
Counter Tracer Example
This tracer counts the total number of times the method is called.
TraceOneMethodOfClass: com.sun.petstore.cart.ShoppingCart placeOrder BlamedMethodTraceIncrementor "Petstore|ShoppingCart|placeOrder:Total Order Count"
Combined Counter Tracers Example
These tracers combine incrementor and decrementer Tracers to keep a running count.
TraceOneMethodOfClass: com.sun.petstore.account.LoginEJB login MethodTraceIncrementor "Petstore|Account:Logged In Users"
TraceOneMethodOfClass: com.sun.petstore.account.LogoutEJB logout MethodTraceDecrementor "Petstore|Account:Logged In Users"
Advanced single-metric tracers
Directives and tracers track methods, classes, and sets of classes. A single-metric tracer reports a specific metric for a specific method, which is the smallest unit that Introscope can track. Single-metric tracers can be created in several ways: through the method signature, by substituting keywords, or by manipulating the metric name parameters.
Signature Differentiation
You can apply tracers to a method based on the method signature.
To trace a single instance of a method with a specific signature, append the signature to the method name.  Include the return type that is specified using the internal method descriptor format.
For example,
myMethod(Ljava/lang/String;)V
traces the instance of the method with a string argument and void return type.
For complete information about this format, see the
Sun Java Virtual Machine Specification
.
Metric Name Keyword-Based Substitution
Keyword-based substitution allows runtime substitution of values into the metric name.
The parameters in the metric name in the tracer are substituted at runtime for the actual values into the metric name. This feature can be used with any directive.
  • {method}
    Name of the method being traced
  • {classname}
    Runtime class name of the class being traced
  • {packagename}
    Runtime package name of the class being traced
  • {packageandclassname}
    Runtime package and class name of the class being traced
If Introscope processes a class that does not have a package, it replaces
{packagename}
with the string
"<Unnamed Package>"
.
Keyword-Based Substitution: Example 1
A tracer has this metric name in the .pbd file:
{packagename}|{classname}|{method}:Response Time (ms)
The tracer is applied to method
myMethod
with a runtime class of
myClass
that is in package
myPackage
.
The resulting metric name is:
myPackage|myClass|myMethod:Response Time (ms)
Keyword-Based Substitution: Example 2
A tracer has this metric name in the .pbd file:
{packageandclassname}|{method}:Response Time (ms)
The tracer is applied to the same (
myMethod)
 method.
The resulting metric name is:
myPackage.myClass|myMethod:Response Time(ms)
Notice the difference the resulting metric name in Examples 1 and 2. In Example 1, the pipe symbol | separates
myPackage
and
myClass
. In Example 2, the period symbol
.
separates 
myPackage
 and
myClass
.
Metric-name-based Parameters
A single-method tracer can create a metric name that is based on parameters that are passed to a method.  Use this format to create a metric name in this situation:
TraceOneMethodWithParametersOfClass: <class-name> <method> <tracer-name> <metric-name>
Introscope uses the 
TraceOneMethodWithParametersOfClass
 keyword to pass the parameters to the method.
You can use parameters in the metric name by substituting the parameter values for placeholder strings in the metric name. The placeholder strings to use are hash mark {
#
} where # is the index of the parameter to substitute. The indexes start counting at zero. Any number of parameter substitutions can be used in any order. All parameters are converted to strings before substitution into the metric name. Object parameters other than strings should be used with caution because they are converted using the
toString()
method.
If you are unclear about what string the parameter is converted to, do not use it in the metric name.
Metric Name Based on Parameter Example
A website uses a class-name order, with a method named process. The method has parameters for different kinds of orders, either book or music.
You can create a tracer as shown in this example:
TraceOneMethodWithParametersOfClass: order process(Ljava/lang/string;)V MethodTimer "Order|{0}Order:Average Response Time (ms)"
This tracer could produce metrics that are shown in this example:
Order
BookOrder
  • Average Response Time (ms)
MusicOrder
  • Average Response Time (ms)
You can also use the TraceOneMethodWithParametersIfInherits keyword.
Skip Directives
Certain packages, classes, or methods can be skipped by AutoProbe or ProbeBuilder by using skip directives. By default, the Java agent and fundamental Java classes and packages are skipped by AutoProbe or ProbeBuilder.
When your skip directive contains references to classes whose names include the
-
(hyphen) character, the classes are not skipped. A cosmetic error appears in the log, and functionality is not affected.
The
-
(hyphen) character is known to be illegal in class names. Class names are likely to contain hyphens when the names have been dynamically generated.
Counting Object Instances
The
InstanceCounts
tracer group counts the number of instances of the particular object types that are associated with it. Any instances that are explicitly allocated in your code are counted. Subtypes are also be counted. Objects that are created through different mechanisms, such as deserialization or cloning, might not be counted. Tracing using this tracer group could potentially incur incremental performance (and memory) impact, depending entirely on the number of instances counted.
Turning on InstrumentPoint Directives
The keyword
InstrumentPoint
identifies the following types of directives:
  • Directives that trace exceptions
  • Directive that causes agent initialization when the application starts up (instead of when the first Probe is run)
Exceptions
The following directives are used to turn on tracing of exceptions either where thrown or caught. These directives can cause performance degradation, so they are turned off by default. To turn either of these directives on, uncomment the appropriate line:
#InstrumentPoint: ThrowException
#InstrumentPoint: CatchException
Agent initialization
The agent initialization instrument-point directive does not cause extra overhead and is turned on by default in both full and typical PBD sets.
InstrumentPoint: AgentInitialization
When you use multiple ProbeBuilder Directive files, any settings that are turned on in any file take effect. Examples of settings are tracer groups, skips, instrumentPoint directives, and custom method traces.
Combining custom tracers
You can use multiple tracers that affect the same metric, in effect combining them. You use multiple tracers most commonly with incrementors and decrementers.
This example creates a metric named Total Purchases. You use the 
cart
class and 
buyBook
and
buyCD
 methods to create the following tracers:
TraceOneMethodOfClass cart buyBook PerIntervalCounter "Total Purchases"
TraceOneMethodOfClass cart buyCD PerIntervalCounter "Total Purchases"
This use of multiple tracers increments the Total Purchases metric when someone buys a piece of merchandise.
Java Annotations
Introscope allows the use of Java 1.6 annotations when creating custom metrics.
More information
: About Java annotations, see the Java developer website.
Use
IdentifyAnnotatedClassAs
to place the class in a tracer group, and then use
TraceAllMethodsIfFlagged
directives to instrument the methods in the class. For example:
SetFlag: AnnotationTracing TurnOn: AnnotationTracing
IdentifyAnnotatedClassAs: com.test.MyAnnotation AnnotationTracing
TraceAllMethodsIfFlagged: AnnotationTracing BlamePointTracer "Target|MyTarget|{classname}"
In the example,
com.test.MyAnnotation
is the annotation name. When creating your own annotations, use a term in your code. Classes containing the annotation name are identified.