OpenTracing Support

DX APM
OpenTracing Support reports metrics and transaction traces from applications instrumented with OpenTracing-compliant tracers. Typically, the applications are microservices applications.
OpenTracing is an open-source, vendor-neutral API Cloud Native Computing Foundation (CNCF) project. OpenTracing includes specifications, and provides tracers to instrument microservices applications. OpenTracing tracers are written for a variety of languages including Java, PHP, Python, and Go.
Jaeger is an open-source CNCF project that is a transaction performance monitoring system for distributed systems. Jaeger provides OpenTracing-compliant tracers.
Jaeger uses terms defined by CNCF to describe the workflow and performance reporting components. The terms span, tag, trace, and span context are particularly helpful for you to understand. See the OpenTracing documentation for the definitions.
OpenTracing Support collects span data from Jaeger tracers and sends the data to
DX APM
, which displays the associated metrics and transaction traces. OpenTracing Support allows you to collect information about performance issues for microservices applications instrumented with Jaeger. OpenTracing Support is a type of Infrastructure Agent extension. OpenTracing Support works with Jaeger tracing, which provides instrumentation for custom applications, libraries, infrastructure services written in Java.
Install OpenTracing Support
OpenTracing Support is available from the Universal Monitoring Agent, from the APM Download Dialog, and from APM Command Center. We recommend that you install using the Universal Monitoring Agent.
Install OpenTracing Support Using the Universal Monitoring Agent
Configure Your Jaeger-Instrumented Application (UMA)
You configure the Jaeger tracer to report traces to the Infrastructure Agent.
Follow these steps:
  1. Set the environment variable as shown:
  2. JAEGER_ENDPOINT=http://<apmia-host>:31313/8888
  3. Restart the instrumented application.
Install OpenTracing Support from the APM Download Dialog
You install the Infrastructure Agent to deploy OpenTracing Support.
Download and Install the Infrastructure Agent
After you get access to
DX APM
, you download and extract the Infrastructure Agent containing OpenTracing Support.
Follow these steps:
1. Log in to
DX SaaS
.
2. Click
Open
on the
DX Application Performance Management
box.
The
DX APM
home page appears.
3. Click the Agent icon in the lower section of the navigation pane.
4. On the
Select Agent to Download
page, click the
Download Agent
button.
5. On the
Select Agent to Download
page, select the tab corresponding to your operating system.
6. Select
Open Tracing
.
DX APM
downloads the package to your computer.
Run the Infrastructure Agent
You install the Infrastructure Agent to deploy OpenTracing Support.
Follow these steps:
1. Navigate to the
apmia
directory.
2. Run the appropriate command to install the Infrastructure Agent:
UNIX: ./APMIACtrl.sh install
Windows: APMIACtrl.bat install
You can review the Infrastructure Agent logs in the <Infrastructure_Agent_Home>/logs directory.
These log files are in the logs directory:
  • AutoProbe.log
  • IntroscopeAgent.log
  • APMIAgentConsole.log
Configure Your Jaeger-Instrumented Application (Infrastructure Agent)
You configure the Jaeger tracer to report traces to the Infrastructure Agent.
Follow these steps:
  1. Set the environment variable as shown:
    JAEGER_ENDPOINT=http://<apmia-host>:8080/open-tracing/spans/ingest
  2. Restart the instrumented application.
Configure OpenTracing Support
You can configure optional properties to fine-tune OpenTracing Support.
Follow these steps:
1. Navigate to the <Agent_Home>;/releases/<version>/extensions/open_tracing directory.
2. Open the bundle.properties file.
3. (Optional) Configure these properties:
introscope.agent.opentracing.trace.filtering.enabled
This property determines whether the OpenTracing Support Agent filters transaction traces. When set to true, DX APM reports only sample, error, discovery, and automatic or custom-set response time threshold traces. All other traces are filtered out. When set to false, DX APM reports all traces.
Values:
true, false
Default:
false (no filtering)
Requires application restart
: Yes
introscope.agent.opentracing.vertex.attributes.regex.list
This property is a list of regular expressions used to display specific span data points on the map. This property is used when given span has a list of tags. When a span tag name matches any of the regular expression, the tag is shown as an attribute in the data point corresponding to span.
Values:
regular expression characters
Default:
*
Requires application restart:
Yes
Introscope.agent.opentracing.agent.agingPeriod.minutes
This property determines the number of minutes after which data for a non-reporting Virtual Agent ages out, and does not appear on the map.
Values:
integer > 0
Default:
1440 (1 day)
Requires application restart
: Yes
4. Save and close the file.
OpenTracing Virtual Agents
OpenTracing creates a Virtual Agent for each Jaeger-instrumented application. Here is the path in the metrics browser:
SuperDomain|<traced-application-host>|opentracing|<service-name>Agent
This figure shows the three OpenTracing Support metrics for the
systemks12
host, which uses OpenTracing to instrument the
authenticatorAgent
application. Notice that you can view data about backends, frontends, and spans.
OpenTracing Support Metrics for Example Host
OpenTracing Support metrics for example host
OpenTracing Metrics
OpenTracing gathers and reports these three metrics from every span collected:
• Average Response Time (ms)
• Responses Per Interval
• Errors Per Interval
OpenTracing Support reports distributed application components. OpenTracing Support reports frontends, HTTP backends, database backends, and raw spans. Here are the paths in the metric browser:
Frontends Path
Frontends|Apps|<service-name>|<operation-name>
HTTP Backend Path Backends|WebService at <client.proto>_//<client.hostname>_<client.port>
Database Backend Path Backends|<db-instance-name> on <db-host> (<db-type>)
Raw Span Path Spans|Apps|<service-name>|<operation-name>
OpenTracing Support Transaction Traces
OpenTracing Support creates transaction traces for transactions executed by Jaeger-instrumented applications. Multiple traces are correlated when the traces are part of a cross-process distributed transaction. In this example transaction trace figure, you can see the transaction calls that the greeting application made in a workflow.
OpenTracing Support Example Transaction Trace
OpenTracing Support example transaction trace
The transaction trace details include span tags and logs, as shown in this example figure.
OpenTracing Support Example Transaction Trace Details
OpenTracing Support Example Transaction Trace Details
In this figure of an example Error transaction trace, you can see that the
authenticate
call took 5048 milliseconds before timing out. The timed-out call caused the error.
OpenTracing Support Example Error Transaction Trace
OpenTracing Support Example Error Transaction Trace
OpenTracing Support Displayed in the Map
OpenTracing Support workflow in the Application Layer displays correlation between the vertices, as shown in this figure.
OpenTracing Support Workflow in the Application Layer
OpenTracing Support Workflow in the Application Layer
OpenTracing Support workflow in the Infrastructure layer displays correlation with data from a number of other Infrastructure Agent extensions. Data is correlated with that from Database Monitoring, Host Monitoring, Docker Monitoring, and Kubernetes Monitoring. This figure shows OpenTracing Support correlated with data from an Oracle database.
OpenTracing Support Workflow in the Infrastructure Layer
OpenTracing Support Workflow in the Infrastructure Layer
OpenTracing Performance Overhead
The number of spans that OpenTracing Support can handle depends on your computer capacity. A computer with more resource capacity can handle higher load. A 4 vCPU @ 1.7GHz and 8GB RAM virtual machine can handle about 1 million spans per 15-second interval, which is equivalent to 6000 transactions per second. If your environment handles more than one million spans per interval, add more CPU. Add one additional CPU for each 200,000 to 300,000 spans per interval.
OpenTracing Support Troubleshooting
Symptom
: I do not see metrics and transaction traces. What could the problem be?
Solution
: The issue could be caused by several problems. Check the following:
• Examine the ingestion endpoint GET and Response values, which should look like these:
GET http://<apmia-host>:8080/open-tracing/spans/ingest Response “Open Tracing Servlet GET Method”Response “Open Tracing Servlet GET Method”
• Review the OpenTracing Sustainability Metrics
Navigate to <apmia-host>|Infrastructure|Agent|Agent Stats|Sustainability|Open Tracing in the metric browser.
For all of these supportability metrics a value greater than 0 indicates that the OpenTracing Agent is receiving span data. A value of 0 indicates a problem; the OpenTracing Agent is not receiving span data.
o Virtual Agent Count
o Spans Received Total
o Spans Received Per Interval
* Review the Infrastructure Agent Debug Log
Follow these steps:
1. Navigate to the <Infrastructure_Agent_Home>/logs directory.
2. Open the
IntroscopeAgent.log
file.
3. Set the log configuration as shown:
log4j.logger.IntroscopeAgent.OpenTracingService=DEBUG, logfile log4j.logger.IntroscopeAgent.OpenTracingService=TRACE#com.wily.util.feedback.Log4JSeverityLevel, logfile
4. Exercise the Jaeger-instrumented application.
5. Look for messages similar to these key messages:
o
[INFO] [IntroscopeAgent.OpenTracingService] Open Tracing Service starting.
This message indicates that OpenTracing Support has been loaded and started.
o
[TRACE] [IntroscopeAgent.OpenTracingService] collector received post request from remote host: 192.170.65.213.
This message indicates that the OpenTracing Support Agent has received a span ingestion request from a Jaeger-instrumented application running on host 192.170.65.213.
o
[INFO] [IntroscopeAgent.OpenTracingService] new Agent route authenticator@systemks12 has been added.
This message indicates that the OpenTracing Support Agent has added new virtual agent with a given name of
application name@host
. In this example, the virtual agent is
authenticator@ systemks12
. The OpenTracing Support Agent is getting spans from the
authenticator
application running on host
systemks12
.
• Examine your Jaeger tracer configuration.
Carefully review the Jaeger 1.17 tracer configurations and Jaeger Troubleshooting for errors in your configuration or other likely solutions.