Work with ProbeBuilder Directives

The agent provides monitoring for many components of .NET applications by default. You generally instrument at least some classes and methods that are specific to your applications. This section provides an introduction to working with the ProbeBuilder Directive keywords and creating custom PBD files.
apmdevops106
The agent provides monitoring for many components of .NET applications by default. You generally instrument at least some classes and methods that are specific to your applications. This section provides an introduction to working with the ProbeBuilder Directive keywords and creating custom PBD files.
2
2
PBDs and PBLs only support ASCII characters. Placing other characters (such as Unicode characters) in PBDs or PBLs could result in problems.
Adding Classes to an Existing Tracer Group
You can turn on tracing for a particular class by adding the class to an existing tracer group. To identify a class as being part of a tracer group, use one of the identify keywords.
For example, to add the class
System.EnterpriseServices.ServicedComponent
to the tracer group
ServicedComponentTracing
:
IdentifyClassAs:
System.EnterpriseServices.ServicedComponent
ServicedComponentTracing
Creating Custom Tracers
You can further refine your metric collection by creating custom .pbd files. Creating custom directives, by creating tracers to track application-specific measurements, requires the use of specific syntax and keywords.
To write custom tracers, you must define:
  • The directive type (indicating generically how many classes) or methods to trace).
  • The specific classes or methods to trace.
  • The type of information to trace in the classes or methods (for example, a time, a rate, or a count).
  • The fully qualified metric name (including the resource path) under which to present this information.
Once a custom .pbd is created, Introscope treats as an out-of-the-box .pbd. You can set alerts on the metrics, save the metrics to SmartStor, and use the metrics when you create custom dashboards.
Be sure to choose methods to trace carefully, as more methods traced means more overhead.
Common Custom Tracer Example
A
BlamePointTracer
is the most commonly used tracer. This tracer generates five separate metrics for associated methods or classes:
  • Average Response Time (ms)
  • Concurrent Invocations
  • Errors Per Interval
  • Responses Per Interval
  • Stall Count
Here is an example of a
BlamePointTracer
. A
BlamePointTracer
has been set for a method named
search
in class
petshop.catalog.Catalog
. "MSPetShop|Catalog|search" is the name of the node in the Introscope Investigator that the BlamePoint metrics are displayed under:
TraceOneMethodOfClass: petshop.catalog.Catalog search BlamePointTracer "MSPetShop|Catalog|search"
Tracer Syntax
In addition to simple keywords that associate tracers into groups or enable/disable groups, PBD files contain tracer definitions. For Introscope to recognize and process your tracers, you must use specific syntax when constructing custom tracers. A tracer is composed of a directive and information about the method or class to trace. You provide the tracer information using this format:
<directive>: [arguments]
[arguments]
is a list and directive-specific. Arguments that are used in trace directives include <
Tracer-Group
>, <
class
>, <
method
>, <
Tracer-name
>, and <
metric-name
>.
Depending on the directive that is used, only a subset of these properties are required.
<directive>
Six main directives are available for custom tracing:
  • TraceOneMethodOfClass
    -- traces a specified method in the specified class.
  • TraceAllMethodsOfClass
    -- traces all methods in the specified class.
  • TraceOneMethodIfInherits
    -- traces one method in all direct subclasses or direct interface implementations of the specified class or interface.
  • TraceAllMethodsIfInherits
    -- traces all methods in all direct subclasses or direct interface implementations of the specified class or interface.
Only concrete, implemented methods can be traced and can report metric data while running. An abstract method specified in a custom tracer results in no metric data being reported.
  • TraceOneMethodIfFlagged
    -- traces one method if the specified class is included in a tracer group that has been enabled with the
    TurnOn
    keyword.
  • TraceAllMethodsIfFlagged
    -- traces all methods if the specified class is included in a tracer group that has been enabled with the
    TurnOn
    keyword.
<Tracer-Group>
The group to which the tracer is associated.
<class>
A fully qualified class or interface name to trace. Fully qualified classes include the full assembly name of the class and the name, for example:
[MyAssembly]com.mycompany.myassembly.MyClass
The assembly name must be enclosed in [] brackets.
<method>
The method name (for example,
MyMethod
)
OR
The full method signature with return type and parameters, for example,
myMethod;[mscorlib]System.Void([mscorlib]System.Int32)
.
<Tracer-name>
Specifies the tracer type to be used. For example,
BlamePointTracer
. See Tracer names for descriptions of tracer names and what they trace.
<metric-name>
Controls how the collected data is displayed in the Introscope Workstation.
Here are examples of three ways to specify the name and location of a metric at different  metrics tree levels.
  • metric-name
    -- the metric appears immediately inside the agent node.
  • resource:metric-name
    -- the metric appears inside one resource (folder) below the agent node.
  • resource|sub-resource|sub-sub-resource:metric-name
    -- the metric appears more than one resource (folder) level deep below the agent node. Use pipe characters (|) to separate the resources.
Tracer Names
Here are tracer names and what they trace.
BlamePointTracer
Provides a standard set of metrics including average response time, per interval counts, concurrency, stalls, and errors for a blamed component.
ConcurrentInvocationCounter
Reports the number of times a method has started but not yet finished. The result is reported under the metric name specified in the tracer, <metric-name>, in the Investigator tree. An example use of this tracer would be counting the number of simultaneous database queries.
DumpStackTraceTracer
This tracer dumps a stack trace to the instrumented application standard error for methods to which it is applied. The exception stack trace that the Dump Stack Tracer throws is not a true exception. This exception is a mechanism for printing the method stack trace.
This feature is useful for determining call paths to a method.
This feature imposes heavy system overhead. We strongly recommend that you only use this tracer for diagnosis in an environment where a sharp overhead increase is acceptable.
MethodTimer
Average method execution time in milliseconds and reports it under the metric name specified in the tracer,
<metric-name>
,
in the metrics tree.
PerIntervalCounter
Number of invocations per interval. This interval changes based on the view period of the consumer of the data (for example, the Investigation View pane). It is reported under the metric name specified in the tracer,
<
metric-name
>
, in the Investigator tree.
Custom Method Tracer Examples
Custom tracers can contain metric names with spaces in them. When using spaces in your custom metric names, CA Technologies recommends putting quotes ("") around all metric names.
Do not place quotation marks around class names. The quotatin marks cause custom tracers to malfunction. For example:
  • Correct
    IdentifyClassAs: My.Name.Space.MyClass MyTracers
  • Incorrect
    IdentifyClassAs: "My.Name.Space.MyClass" MyTracers
If you create a metric name that contains a class name, you must use quotes around the whole metric name. Metric names are allowed to have spaces, and all spaces in metric names must be contained within quotes. For example, the metric name "{classname}|Test One Node" would be represented as follows:
  • Correct
    TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer "{classname}|Test One Node"
  • Incorrect
    TraceOneMethodIfFlagged: MyTracers AMethod BlamePointTracer {classname}|Test One Node
The following sections provide examples of method tracers. In the following example, quotes ("") are used around the metric names. CA Technologies highly recommends putting quotes around all metric names when you create custom metric names.
Average Tracer Example
This tracer tracks the average execution time of the given method in milliseconds.
TraceOneMethodOfClass: petshop.catalog.Catalog search BlamedMethodTimer "MSPetShop|Catalog|search:Average Method Invocation Time (ms)"
Rate Tracer Example
This tracer counts the number of times the method is called per second, and reports this rate under the specified metric name.
TraceOneMethodOfClass: petshop.catalog.Catalog search BlamedMethodRateTracer "MSPetShop|Catalog|search:Method Invocations Per Second"
Per Interval Counter-Tracer Example
This method tracer counts the number of times the method is called per interval. The tracer reports the per-interval count under the specified metric name.
TraceOneMethodOfClass: petshop.catalog.Catalog search PerIntervalCounter "MSPetShop|Catalog|search:Method Invocations Per Interval"
The monitoring logic in the Enterprise Manager determines the interval, such as the Graph frequency.
The preview pane in the Introscope Investigator defaults to 15-second intervals.
Counter Tracer Example
This tracer counts the total number of times the method is called.
TraceOneMethodOfClass: petshop.cart.ShoppingCart placeOrder BlamedMethodTraceIncrementor "MSPetShop|ShoppingCart|placeOrder:Total Order Count"
Combined Counter Tracer Example
These tracers combine incrementor and decrementor tracers to keep a running count.
TraceOneMethodOfClass: petshop.account.LoginComponent login MethodTraceIncrementor "MSPetShop|Account:Logged In Users"
TraceOneMethodOfClass: petshop.account.LogoutComponent logout MethodTraceDecrementor "MSPetShop |Account:Logged In Users"
Creating Advanced Custom Tracers
The following sections detail creating advanced customs tracers, such as single-metric tracers, skips, and combined custom tracers.
Advanced Single-Metric Tracers
Directives and tracers track methods, classes, and sets of classes. A single-metric tracer reports a specific metric for a specific method, which is the smallest unit that Introscope can track. Single-metric tracers can be created in several ways: through the method signature, by substituting keywords, or by manipulating the metric name parameters.
Signature Differentiation
Tracers can be applied to a method based on the method signature.
To trace a single instance of a method with a specific signature, append the signature to the method name (including return type) specified using the internal method descriptor format.
For example,
myMethod;[mscorlib]System.Void([mscorlib]System.Int32
) traces the instance of the method with an
int
argument and a
void
return type.
Metric Name Keyword-Based Substitution
Keyword-based substitution allows runtime substitution of values into the metric name.
At runtime the parameters in the metric name in the tracer are substituted for the actual values into the metric name. This feature can be used with any directive. The following table lists the parameters and their runtime substitutions:
Parameter
Runtime substitution
{method}
Name of the method being traced
{classname}
Runtime class name of the class being traced
{namespacename}
Runtime namespace name of the class being traced
{namespaceandclassname}
Runtime namespace and class name of the class being traced
{assemblyname}
Name of the assembly being traced.
{fullclassname}
Reports the full class name including the assembly name.
If Introscope processes a class that does not have a
namespace
, it replaces
{namespacename}
with the string
<Unnamed Namespace>.
Keyword-based substitution: Example 1
You provide the metric name for a tracer in the .pbd file is written.
"{namespacename}|{classname}|{method}:Response Time (ms)"
When a tracer is applied to the method
myMethod
with a runtime class of
myClass
that is in
myNamespace
, this is theresulting metric name:
"myNamespace|myClass|myMethod:Response Time (ms)"
Keyword-based substitution: Example 2
A tracer has this metric name in the .pbd file:
"{namespaceandclassname}|{method}:Response Time (ms)"
When  the tracer is applied to the same method shown in example 1, here is the resulting metric name:
"myNamespace.myClass|myMethod:Response Time(ms)"
In this example, a . (period) is used between the
namespace
and
class
instead of the | (pipe symbol) in the first example.
Metric Name-Based Parameters
You can create a single-method tracer that creates a metric name. This metric name is based on parameters that are passed to a method using the
TraceOneMethodWithParametersofClass
keyword, using this format:
TraceOneMethodWithParametersOfClass: <
class
> <
method
> <
Tracer-name
> <
metric-name
>
Parameters can be used in the metric name. Substitute the value of parameters for placeholder strings in the metric name. Use the following string as a place holder:"{#}", where # is the index of the parameter to substitute. The indexes start counting at zero. Any number of parameter substitutions can be used in any order. All parameters are converted to strings before substitution into the metric name. Use object parameters other than strings with caution because they are converted using the
ToString()
method.
If you are unclear about what string the parameter converts to, do not use it in the metric name.
Metric name based on parameters example
A website uses a class named
order
, with a method named
process
. The method has parameters for different kinds of orders, either
book
or
music
.
You can create a tracer as shown in this example:
TraceOneMethodWithParametersOfClass: order process; [mscorlib]System.Void([mscorlib]System.Int32) MethodTimer "Order|{0}Order:Average Response Time (ms)"
This tracer produces the metrics shown in this example:
Order
BookOrder
Average Response Time (ms)
MusicOrder
Average Response Time (ms)
You can also use the
TraceOneMethodWithParametersIfInherits
keyword.
Skip Directives
You can avoid instrumenting specific packages, classes, or methods by using skip directives. Skip directives cause ProbeBuilder to skip over a namespace, class, or assembly.
You can also use reduced matching to limit the number of packages, classes, or methods being instrumented. You reduce matching by replacing the
IdentifyAllClassAs
and
TraceAllMethodsIfFlagged
directives (or one if both are not used) with something less inclusive.
If your skip directive contains references to classes with names including the "-" (hyphen) character, the classes are not skipped, and an error appears in the log. However, this error is cosmetic and functionality is not affected.
As the “-“ character is known as illegal in class names, it is likely that class names contain these characters if the names have been dynamically generated.
Combining Custom Tracers
You can use multiple tracers that affect the same metric, in effect combining them. Combining multiple tracers is most commonly used with incrementors and decrementors.
This example creates a metric named Logged-in User. With a class
user
and methods
l
ogin
and
logout,
create the following tracers:
TraceOneMethodOfClass user login MethodTraceIncrementor "Logged-in Users"
TraceOneMethodOfClass user logout MethodTraceDecrementor "Logged-in Users"
This code increments the metric Logged-in Users when someone logs in, and the code decrements Logged-in Users when someone logs out.
Notes About Specific Tracers
The following identifier and tracer have actions specific to a .NET environment:
  • IdentifyAnnotatedClassAs
    : <attribute-class-name> <Tracer-group>
    Associates all classes annotated with the specified attribute class to the specified tracer group.
    Some classes can be annotated with an attribute class to provide extra functionality to the class. In the example below, an attribute class that is named
    System.EnterpriseServices.Transaction
    is attached to the class
    ServicedComponent
    :
    [Transaction]
    Public class ServicedComponent {
    }
    In a .pbd you would write the following line:
    IdentifyAnnotatedClassAs: System.EnterpriseServices.Transaction MyTracerGroup
    This line identifies all classes that have the
    [Transaction]
    annotation, including
    ServicedComponent
    .
The Introscope .NET Agent does not trace inherited attributes when using this identifier. The .NET Agent does trace attributes applied to the base class.
  • TraceAnnotatedMethodsIfFlagged
    : <Tracer-group> <attribute-class-name> <Tracer-name> <metric-name>
    Traces all methods which are annotated by the specified class for classes associated with the specified tracer group.
Explicit Interface Implementation
The .NET Agent uses explicit interface implementation in .pbd files. When you are tracing a method of a class with the identical name as another method used in another class, explicitly name the method and the interface to which it belongs. For example,
InterfaceA
and
InterfaceB
both have a method named
MethodX.W
hen calling
MethodX
for
InterfaceA
, you must name both the interface and the
InterfaceA.MethodX
method.
Here is an example of tracing a method of a class with an explicit interface implementation:
SetFlag: customInterfaceTracing
TurnOn: customInterfaceTracing
IdentifyInheritedAs: EdgeCaseInterface customInterfaceTracing
TraceOneMethodIfFlagged: customInterfaceTracing EdgeCaseInterface.method2 BlamePointTracer "Interface|{namespaceandclassname}|{method}"
Instrumenting and Inheritance
Introscope does not automatically instrument classes in the deeper levels of a class hierarchy. For example, assume a class hierarchy in which
ClassB
extends
ClassA
, and
ClassC
extends
ClassB
as shown:
Interface\ClassA
ClassB
ClassC
When you instrument
ClassA
,
ClassB
is also instrumented because it explicitly extends
ClassA
. However, Introscope does not instrument
ClassC
because
ClassC
does not explicitly extend
ClassA
. To instrument
ClassC,
you must explicitly identify
ClassC
.
Applying ProbeBuilder Directives
When you are ready to implement a ProbeBuilder Directive file, there are three ways to implement the new files:
Using the hotdeploy Directory
The hotdeploy directory allows Introscope administrators to deploy new directives more quickly and easily, without editing the IntroscopeAgent.profile, and potentially without restarting applications. This ability heightens the need for caution. If your custom PBDs contain invalid syntax, or are configured to collect too many metrics, the impact is felt more quickly. Invalid PBDs can cause
NativeProfiler
to shut off and PBDs that collect too many metrics can affect application performance. To address this situation, we recommend:
  • Testing and validating all directives in QA and performance environments before pushing them out to production environments.
  • Ensuring that your server environment change control process is updated to reflect the new option for deploying PBDs.
When a new PBD is placed in the hotdeploy directory, the .NET Agent automatically deploys this new PBD. The new or changed PBD classes do not affect applications that are already running until the application restarts. When new PBDs are placed in this directory, you do not have to edit the IntroscopeAgent.profile to pick up new or changed PBDs.
To apply .pbds using the hotdeploy directory:
  • Copy custom or modified files (.pbd
    s
    and .pbls) into the
    <Agent_Home>
    \hotdeploy
    directory.
Using the <
Agent_Home
> Directory
To deploy new or changed PBD and PBL files, you must include them in the
introscope.autoprobe.directivesFile
property. You place the files in the same directory as the IntroscopeAgent.profile file or in a directory relative to the location of the IntroscopeAgent.profile file.
If you place files in any other directory, you must specify the full path to the file when setting the
introscope.autoprobe.directivesFile
property.
To deploy new and changed .pbd and .pbl files:
  1. Copy custom or modified files (PBDs
    and .PBLs) into the
    <Agent_Home>
    directory.
  2. Update the
    introscope.autoprobe.directivesFile
    property in the IntroscopeAgent.profile file to include the names of any new files, which are separated by commas.
    For example, add the custom petstore.pbd file to the property:
    introscope.autoprobe.directivesFile=default-full.pbl,petstore.pbd,hotdeploy
  3. Save and close the IntroscopeAgent.profile.
  4. Restart the application or IIS services.
Do not remove existing
.pbl
or
.pbd
files that are defined for the property unless you want to disable the monitoring controlled by the existing .pbl or .pbd files.
Custom Locations and Permissions
In addition to using the hotdeploy directory or the
<Agent_Home>
directory, you can place PBDs in a custom location.
If you place .pbd files in a custom location, you must specify the location of the .pbd files in the IntrosocopeAgent.profile. For example, if you placed the leakhunter.pbd in a custom location on the C: drive, you update the
introscope.autoprobe.directivesFile
property as shown:
introscope.autoprobe.directivesFile=default-full.pbl,C:\\sw\\leakhunter.pbd
When placing .pbds in a custom location, the user starting the IIS process needs appropriate permissions to that custom location. For example,
C:\\sw
in the earlier example. If the user starting the IIS process does not have permissions to this location, an error message is reported in the default domain logs, and the .pbds in the custom location do not take effect.
We highly recommend that you place PBDs in the hotdeploy directory.
Transaction Tracing and Dynamic Instrumentation
When you run a transaction trace from the Introscope Workstation, you can perform dynamic instrumentation. Dynamic instrumentation enables you to select one or more methods to instrument at runtime without creating a PBD file or modifying the IntroscopeAgent,profile. With dynamic instrumentation, you can accomplish the following tasks:
  • View and instrument one, more, or all the called methods belonging to a transaction component.
  • View traces on temporarily instrumented methods.
  • View the metrics collected on temporarily instrumented methods.
  • Make temporary instrumentation permanent.
Dynamic instrumentation is a CPU-intensive operation. CA Technologies highly recommends that you use configurations that minimize the classes that are being instrumented.
Support for the.NET operating environment is limited. For more information, see Transaction Tracing.