Enabling Tracing with OpenTracing

In a distributed API Manager architecture, tracing a message is important to debug and observe a message path. This is known as distributed tracing. OpenTracing allows you to enable distributed tracing for WSO2 API Manager. OpenTracing aims to be an open, vendor-neutral standard for distributed systems instrumentation. It offers a way for developers to follow the thread — to trace requests from beginning to end across touchpoints and understand distributed systems at scale. Open tracing will also help to trace the message and identify the latencies that took place in each process or method. Thereby, open tracing will help you to carry out a time-related analysis.

WSO2 API Manager supports the following types of ways to retrieve instrumented data.

  • Jaeger
  • Zipkin
  • Log

For more information, see Open Tracer Configurations.

Enabling Jaeger Tracing

  1. Copy the following configuration into the deployment.toml file.

    [apim.open_tracer]
    remote_tracer.enable = true
    remote_tracer.name = "jaeger"
    remote_tracer.properties.hostname = "<hostname-of-jaeger-endpoint>"
    remote_tracer.properties.port = "<port-of-jaeger-endpoint>"
    [apim.open_tracer]
    remote_tracer.enable = true
    remote_tracer.name = "jaeger"
    remote_tracer.properties.hostname = "localhost"
    remote_tracer.properties.port = "6831" 
    #6832 can also be used as the port
  2. Start the server.

    After you invoke the APIs you will see the tracing data in Jaeger as follow:

    Distributed tracing jaeger

Enabling Zipkin Tracing

  1. Copy the following configuration into the deployment.toml file.

    [apim.open_tracer]
    remote_tracer.enable = true
    remote_tracer.name = "zipkin"
    remote_tracer.properties.hostname = "<hostname-of-zikin-endpoint>"
    remote_tracer.properties.port = "<port-o-zipkin-endpoint>"
    [apim.open_tracer]
    remote_tracer.enable = true
    remote_tracer.name = "zipkin"
    remote_tracer.properties.hostname = "localhost"
    remote_tracer.properties.port = "9411"
  2. Start the server.

    After you invoke the APIs you will see the tracing data in Zipkin as follow:

Distributed tracing zipkin

Enabling Log Tracing

  1. Copy the following configuration into the deployment.toml file.

    [apim.open_tracer]
    remote_tracer.enable = false
    log_tracer.enable = true
  2. Start the server.

    After you invoke the APIs you will be able to see tracing data in the wso2-apimgt-open-tracing.log in the <API-M_HOME>/repository/logs folder.

Note

  1. There are some limitations in the Jaeger client, which by default uses a UDP sender as mentioned in the Jaeger documentation. If the payload size exceeds 65 KB, spans might get lost in the Jaeger console.
  2. Jaeger sampler types can also play a major role in tracing. Depending on the TPS, the sampler type should be carefully chosen.
  3. In general, before including tracing in production deployments it is essential to look into performance tests and scaling requirements. Refer the Jaeger performance tuning guide for details on how to achieve better performance.

Use Custom Tracer Implementation

In order to demonstrate this functionality, let's take a scenario with the implementation of Elastic APM (Application Performance Monitoring).

  1. Implement the org.wso2.carbon.apimgt.tracing.OpenTracer interface and add your implementation. The getTracer method should contain the generation of the Tracer instance. Also, the getName method should return the tracer name to be configured in the deployment.toml file. In this specific scenario we are naming this tracer elastic. This tracer needs to be loaded as an osgi service using a module activator. The sample project for the elastic APM tracer can be downloaded from here.

  2. Build the maven project and add the jar file to the dropins directory. (API-M_HOME/repository/components/dropins)

  3. Add the following configuration into the deployment.toml file.

    [apim.open_tracer]
    remote_tracer.enable = true
    remote_tracer.name = <custom_tracer_name>
    [apim.open_tracer]
    remote_tracer.enable = true
    remote_tracer.name = "elastic"
  4. Add the elastic opentracer jar file to the lib directory (API-M_HOME/repository/components/lib). It can be downloaded from here.

    Tip

    Elastic opentracing also requires the addition of a java agent. This can be added by altering the startup script. Make sure to check the documentation for the tracer you are using so that such requirements can be satisfied.

  5. Start the server.

    After you invoke the APIs tracing data will be published to elastic APM

    Distributed tracing elastic

Top