Create Custom Tracers
Contents
apmdevops97
Contents
You can further refine your metric collection by creating custom PBD files. Creating custom directives, by creating tracers to track application specific measurements, require the use of specific syntax and keywords. To write custom tracers, you must define:
- The directive type (indicating generically how many class(es) or method(s) to trace)
- The specific class(es) or method(s) to trace
- The type of information to trace in the class(es) or method(s) (for example, a time, a rate, or a count)
- The fully-qualified metric name (including the resource path) under which to present this information
Custom PBDs are stored in the
<Agent_Home>
/core/config/hotdeploy directory. Any PBDs added to this directory will be implemented without having to update or modify the introscope.autoprobe.directivesFile
property in the IntroscopeAgent.profile. If you have enabled dynamic ProbeBuilding, the PBDs in the hotdeploy
directory are picked up live from the folder. No reboot is required.Once a custom PBD is created, Introscope treats it as if it is an out-of-the-box PBD. You can set alerts on the metrics created, save them to SmartStor, or use them in the creation of custom dashboards in the Introscope Workstation.
Choose methods to trace carefully, as more methods traced means more overhead.
Using a custom BlamePointTracer tracer for common metrics
A
BlamePointTracer
is the most commonly used tracer. This tracer generates five separate metrics for associated methods or classes:- Average Response Time (ms)
- Concurrent Invocations
- Errors Per Interval
- Responses Per Interval
- Stall Count
The following is an example of a
BlamePointTracer
. A BlamePointTracer
has been set for a method called search
in class petshop.catalog.Catalog
. PetShop|Catalog|search
is the name of the node under which the BlamePoint metrics will be displayed in the Introscope Investigator.TraceOneMethodOfClass: petshop.catalog.Catalog search BlamePointTracer "PetShop|Catalog|search"
Directive Names and Arguments used in Tracer Syntax
In addition to simple keywords that associate tracers into groups or enable/disable groups, PBD files contain tracer definitions. For Introscope to recognize and process tracers, use a specific syntax when constructing custom tracers. A tracer is composed of a directive and information about the method or class to trace, in the following format:
<directive>: [arguments]
where
[arguments]
is a list, and is directive-specific.Depending on the directive used, only a subset of the parameters are required.
The most common directives to use are the following trace directives:
- TraceOneMethodOfClassTraces a specified method in the specified class.
- TraceAllMethodsOfClassTraces all methods in the specified class.
- TraceOneMethodIfInheritsTraces a specific method in all direct subclasses of the specified class or in all direct interface implementations of the specified interface.The specified class or interface should be the fully qualified name.
- TraceAllMethodsIfInheritsTraces all methods except for constructors and static initializers in all direct subclasses of the specified class or direct implementations of the specified interface. The specified class or interface should be the fully qualified name.
- TraceOneMethodIfFlaggedTraces one method if the specified class is included in a tracer group that has been enabled with theTurnOnkeyword.
- TraceAllMethodsIfFlaggedTraces all methods except for constructors and static initializers if the specified class is included in a tracer group that has been enabled with theTurnOnkeyword.
- TraceComplexMethodsIfFlaggedTraces all public or package-visible non-synthetic methods that call any other method, except constructors and static initializers in the tracer group.
- TraceAllComplexMethodsIfFlaggedTraces all methods, except constructors and static initializers, which call any other method, in the tracer group.
The TraceAll* directives instruments every method of the specified classes. Using these directives can lead to unnecessary overhead in the application and can result in transaction traces that exceed the default 5,000 component limit.
The TraceAllComplex* and TraceComplex* directives instruments classes that you identify in the directive or the associated tracer group.
The TraceOne* directives will only apply instrumentation to the methods that you specify, not any of the other methods that it calls.
Only concrete, implemented methods can be traced and report metric data while running. An abstract method specified in a custom tracer results in no metric data being reported.
The expected syntax for trace directives usually consist of the following arguments:
- <Tracer-Group>The group to which the tracer is associated.
- <class>A fully qualified class or interface name to trace. Fully qualified classes include the full assembly name of the class as well as the name, for example:[MyAssembly]com.mycompany.myassembly.MyClassThe assembly name must be enclosed in [] brackets.
- <method>The method name (for example,MyMethod)ORthe full method signature with return type and parameters (for example,myMethod;[mscorlib]System.Void([mscorlib]System.Int32). For more information about method signatures, see Signature differentiation.)
- <Tracer-name>Specifies the tracer type to be used. For example,BlamePointTracer. For more information about tracer names, see commonly used tracer names.
- <metric-name>Controls how the collected data is displayed in the Introscope Workstation.The following examples describe ways to specify the name and location of a metric at different levels of the metrics tree.metric-name-- the metric appears immediately inside the agent node.resource:metric-name-- the metric appears inside one resource (folder) below the agent node.resource|sub-resource|sub-sub-resource:metric-name-- the metric appears more than one resource (folder) level deep below the agent node. Use pipe characters (|) to separate the resources.
Using Quotes in Custom Tracer Definitions
Custom tracers can contain metric names with spaces in them. When using spaces in your custom metric names, CA Technologies recommends putting quotes ("") around all metric names.
Do not place quotes around class names. This causes the custom tracers to malfunction. For example:
Correct
IdentifyClassAs: MyClass MyTracers
Incorrect
IdentifyClassAs: "MyClass" MyTracers
If you create a metric name that contains a class name, you must use quotes around the whole metric name. Metric names are allowed to have spaces, and all spaces in metric names must be contained within quotes. For example, the metric name "{classname}|Test One Node" should be represented as follows:
Correct
TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer"{classname}|Test One Node"
Incorrect
TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer {classname}|Test One Node
Introscope does not monitor classes having invalid class file names. For example, in the class file name:
org/jboss/seam/example/seambay/AuctionImage$JaxbAccessorM_getData_setData_[B:
The
_[B:
causes the class file name to be invalid. You cannot use an open square brackets ([) as part of the Java class file name. When Introscope encounters such classes having invalid class names, it fails to instrument them and reports them as an error message in the agent logs.The following sections are examples of method tracers. In the following example, quotes ("") are used around the metric names. CA Technologies highly recommends putting quotes around all metric names when you create custom metric names.
Commonly Used Tracer Names and Examples
The following list describes the most commonly used tracer names and what they trace.
- BlamePointTracerProvides a standard set of metrics including average response time, per interval counts, concurrency, stalls, and errors for a blamed component.
- ConcurrentInvocationCounterReports the number of times a method has started but not yet finished. The result is reported under the metric name specified in the tracer,<metric-name>, in the Investigator tree. An example use of this tracer would be counting the number of simultaneous database queries.
- DumpStackTraceTracerDumps a stack trace to the instrumented application's standard error for methods to which it is applied. The exception stack trace thrown by the Dump Stack Tracer is not a true exception -- it is a mechanism for printing the method stack trace.This feature is useful for determining call paths to a method.This feature imposes heavy system overhead. It is strongly recommended that this tracer only be used in a diagnostic context where a sharp increase in overhead is acceptable.
- MethodCPUTimerAverage CPU time (in milliseconds) used during method execution and reports it under<metricname>in the metrics tree.Note:This tracer requires a platform monitor on the supported platform.
- MethodTimerAverage method execution time in milliseconds and reports it under the metric name specified in the tracer,<metric-name>,in the metrics tree.
- PerIntervalCounterNumber of invocations per interval. This interval will change based on the view period of the consumer of the data (for example, the View pane in the Investigator). It is reported under the metric name specified in the tracer,<metric-name>, in the Investigator tree.
Average Tracer Example
This tracer tracks the average execution time of the given method in milliseconds.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search BlameMethodTimerHC "Petstore|Catalog|search:Average Method Invocation Time (ms)"
Rate Tracer Example
This tracer counts the number of times the method is called per second, and reports this rate under the specified metric name.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search BlamedMethodRateTracer "Petstore|Catalog|search:Method Invocations Per Second"
Per interval counter tracer example
This method tracer counts the number of times the method is called per interval, and reports the per interval count under the specified metric name.
TraceOneMethodOfClass: com.sun.petstore.catalog.Catalog search PerIntervalCounter "Petstore|Catalog|search:Method Invocations Per Interval"
The interval is determined by the monitoring logic in the Enterprise Manager, such as the Graph frequency.
The preview pane in the Investigator defaults to 15 second intervals.
Counter Tracer Example
This tracer counts the total number of times the method is called.
TraceOneMethodOfClass: com.sun.petstore.cart.ShoppingCart placeOrder BlamedMethodTraceIncrementor "Petstore|ShoppingCart|placeOrder:Total Order Count"
Combined Counter Tracers Example
These tracers combine incrementor and decrementor Tracers to keep a running count.
TraceOneMethodOfClass: com.sun.petstore.account.LoginEJB login MethodTraceIncrementor "Petstore|Account:Logged In Users"TraceOneMethodOfClass: com.sun.petstore.account.LogoutEJB logout MethodTraceDecrementor "Petstore|Account:Logged In Users"
Advanced single-metric tracers
Directives and tracers track methods, classes, and sets of classes. A single-metric tracer reports a specific metric for a specific method, which is the smallest unit that Introscope can track. Single-metric tracers can be created in several ways: through the method signature, by substituting keywords, or by manipulating the metric name parameters.
Signature Differentiation
Tracers can be applied to a method based on the method signature.
To trace a single instance of a method with a specific signature, append the signature to the method name (including return type) specified using the internal method descriptor format.
For example,
myMethod(Ljava/lang/String;)V
traces the instance of the method with a string argument and void return type.For complete information about this format, see the
Sun Java Virtual Machine Specification
.Metric Name Keyword-Based Substitution
Keyword-based substitution allows runtime substitution of values into the metric name.
The parameters in the metric name in the tracer are substituted at runtime for the actual values into the metric name. This feature can be used with any directive.
- {method}Name of the method being traced
- {classname}Runtime class name of the class being traced
- {packagename}Runtime package name of the class being traced
- {packageandclassname}Runtime package and class name of the class being traced
If Introscope processes a class that does not have a package, it replaces {packagename} with the string "<Unnamed Package>".
Keyword-Based Substitution: Example 1
If the metric name for a tracer in the pbd file is:
"{packagename}|{classname}|{method}:Response Time (ms)"
and the tracer is applied to method myMethod with a runtime class of myClass that is in package myPackage, the resulting metric name would be:
"myPackage|myClass|myMethod:Response Time (ms)"
Keyword-Based Substitution: Example 2
If a tracer with a metric name in the .pbd file of
"{packageandclassname}|{method}:Response Time (ms)"
was applied to the same method, the resulting metric name would be
"myPackage.myClass|myMethod:Response Time(ms)"
The . between the package and class in Example 2 instead of the | in Example 1.
Metric-name-based Parameters
You can create a single-method tracer that creates a metric name based on parameters passed to a method using the
TraceOneMethodWithParametersOfClass
keyword, using this format:TraceOneMethodWithParametersOfClass: <class-name> <method> <tracer-name> <metric-name>
Parameters can be used in the metric name. This is accomplished by substituting the value of parameters for placeholder strings in the metric name. The placeholder strings to use are "{#}" where # is the index of the parameter to substitute. The indices start counting at zero. Any number of parameter substitutions can be used in any order. All parameters are converted to strings before substitution into the metric name. Object parameters other than strings should be used with caution because they are converted using the toString() method.
If you are unclear about what string the parameter will be converted to, do not use it in the metric name.
Metric Name Based on Parameter Example
A website uses a class named order, with a method named process. The method has parameters for different kinds of orders, either book or music.
You can create a tracer as shown in this example:
TraceOneMethodWithParametersOfClass: order process(Ljava/lang/string;)V MethodTimer "Order|{0}Order:Average Response Time (ms)"
This tracer could produce metrics shown in this example:
Order
BookOrder
- Average Response Time (ms)
MusicOrder
- Average Response Time (ms)
You can also use the TraceOneMethodWithParametersIfInherits keyword.
Skip Directives
Certain packages, classes, or methods can be skipped by AutoProbe or ProbeBuilder by using skip directives. By default, the Java agent and fundamental Java classes and packages are skipped by AutoProbe or ProbeBuilder.
If your skip directive contains references to classes whose names include the "-" (hyphen) character, the classes are not skipped, and an error appears in the log. However, this error is cosmetic and functionality is not affected.
As the “-“ character is well known as illegal in class names, it is likely that class names contain these characters if the names have been dynamically generated.
Counting Object Instances
The InstanceCounts tracer group counts the number of instances of the particular object types associated with it. Any instances explicitly allocated in your code will be counted. Subtypes will also be counted. Objects created through different mechanisms, such as deserialization or cloning, might not be counted. Tracing using this tracer group could potentially incur incremental performance (and memory) impact, depending entirely on the number of instances counted.
Turning on InstrumentPoint Directives
The keyword
InstrumentPoint
identifies the following types of directives:- Those that trace exceptions
- Those that causes agent initialization when the application starts up (instead of when the first Probe is run)
Exceptions
The following directives are used to turn on tracing of exceptions either where thrown or caught. They can cause performance degradation so they are not turned on by default. To turn either of these on, uncomment the appropriate line:
#InstrumentPoint: ThrowException#InstrumentPoint: CatchException
Agent initialization
The agent initialization instrument point directive does not cause additional overhead and is turned on by default in both full and typical PBD sets.
InstrumentPoint: AgentInitialization
If multiple ProbeBuilder Directive files are used, any settings (such as tracer groups, Skips, InstrumentPoints, Custom Method Tracers) turned on in any file take effect.
Combining custom tracers
You can use multiple tracers that affect the same metric, in effect combining them. This is most commonly used with incrementors and decrementors.
This example creates a metric named
Total Purchases
. With a class cart
and methods buyBook and buyCD, create the following tracers:TraceOneMethodOfClass cart buyBook PerIntervalCounter "Total Purchases"TraceOneMethodOfClass cart buyCD PerIntervalCounter "Total Purchases"
This increments the metric
Total Purchases
when someone buys a piece of merchandise.Java Annotations
Introscope allows the use of Java 1.6 annotations when creating custom metrics.
For more information about Java annotations, see the Java developer website.
Use IdentifyAnnotatedClassAs to place the class in a tracer group, and then use TraceAllMethodsIfFlagged directives to instrument the methods in the class. For example:
SetFlag: AnnotationTracing TurnOn: AnnotationTracingIdentifyAnnotatedClassAs: com.test.MyAnnotation AnnotationTracingTraceAllMethodsIfFlagged: AnnotationTracing BlamePointTracer "Target|MyTarget|{classname}"
In the example, com.test.MyAnnotation is the annotation name. When creating your own annotations, use a term in your code. Classes containing the annotation name are identified.