Setting up Cloud Native Observability on a VM

Follow the instructions given below to set up a cloud native observability solution for your Micro Integrator (MI) deployment in a VM environment.

You need to start with the minimum deployment, which enables metric monitoring. Once you have set up the minimum deployment, you can add log processing and message tracing capabilities to your solution.

Step 1 - Set up the minimum deployment

The minimum cloud native observability deployment requires Prometheus and Grafana. The Micro Integrator uses Prometheus to expose its statistics to Grafana. Grafana is used to visualize the statistics.

Step 1.1 - Set up Prometheus

Follow the instructions below to set up the Prometheus server:

  1. Download Prometheus from the Prometheus site.

    Tip

    Select the appropriate operating system and the architecture based on your operating system and requirements.

  2. Extract the downloaded file and navigate to that directory.

    Info

    This directory is referred to as <PROMETHEUS_HOME> from hereon.

  3. Open the <PROMETHEUS_HOME>/prometheus.yml file, and in the scrape_configs section, add a configuration as follows:

    global:
      scrape_interval:     15s 
      evaluation_interval: 15s 
    
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']
      - job_name: esb_stats
        metrics_path: /metric-service/metrics
        static_configs:
         - targets: ['localhost:9201']

    Note

    • Do not add or remove spaces when you copy the above configuration to the prometheus.ymal file.
    • In the targets section, you need to add your IP address and the port in which you are running the Micro Integrator server.
  4. To start the Prometheus server, open a terminal, navigate to <PROMETHEUS_HOME>, and execute the following command:

    ./prometheus

    When the Prometheus server is successfully started, you will see the following log:

    Server is ready to receive web requests.

Step 1.2 - Set up Grafana

Follow the instructions below to set up the Grafana server:

  1. Download and install Grafana.

    Tip

    Follow the instructions (for your operating system) on the Grafana website.

  2. Start you Grafana server.

    Tip

    The procedure to start Grafana depends on your operating system and the installation process. For example, If your operating system is Mac OS and you have installed Grafana via Homebrew, you start Grafana by issuing the brew services start grafana command.

  3. Access the Grafana UI from the localhost:3000 URL.

  4. Sign in using admin as both the username and the password.

Step 1.3 - Import dashboards to Grafana

The Micro Integrator provides pre-configured Grafana dashboards in which you can visualize MI statistics.

You can directly import the required dashboards to Grafana using the dashboard ID:

  1. Go to Grafana labs.
  2. Select the required dashboard and copy the dashboard ID.
  3. Provide this ID to Grafana and import the dashboard.
  4. Repeat the above steps to import all other Micro Integrator dashboards.

These dashboards are provided as JSON files that can be manually imported to Grafana. To import the dashboards as JSON files:

  1. Go to Grafana labs, select the required dashboard and download the JSON file.
  2. Sign in to Grafana, click the + icon in the left pane, and then click Import.

    The Import dialog box opens as follows.

    Import Dashboards dialog box

  3. Click Upload.json file. Then browse for one of the dashboards that you downloaded as a JSON file.

  4. Repeat the above two steps to import all the required dashboards that you downloaded and saved.

Step 1.4 - Set up the Micro Integrator

To enable observability for the Micro Integrator servers, add the following Synapse handler to the deployment.toml file (stored in the <MI_HOME>/conf/ folder).

[[synapse_handlers]]
name="CustomObservabilityHandler"
class="org.wso2.micro.integrator.observability.metric.handler.MetricHandler"
After applying the above change, you can start the Micro Integrator with the following JVM property:
-DenablePrometheusApi=true

Step 2 - Optionally, integrate the Log Processing add-on

Once you have successfully set up the minimum deployment, you need to set up the log processing add-on to process logs. To achieve this, you can use Grafana Loki-based logging stack.

A Loki-based logging stack consists of three components:

  • fluentBit is the agent that gathers logs and sends them to Loki.
  • loki is the main server that stores logs and processes queries.
  • Grafana queries and displays the logs.

Follow the steps below to set up Fluent Bit and Grafana Loki:

Step 2.1 - Set up Fluent Bit

Follow the instructions below to set up Fluent Bit:

  1. Download Fluent Bit.

  2. Extract the downloaded file.

    Tip

    The directory is referred to as <FluentBit_Home> from hereon.

  3. Create the following files and save them with the given extension in a preferred location. You can use any text editor of your choice.

    Info

    In this example, the files are saved in the <HOME_DIRECTORY>/conf directory.

    • labelmap.json file

      {
          "instance": "instance",
          "log_level": "log_level",
          "service": "service"
       }
    • parsers.conf file

      [PARSER]
              Name        observability
              Format      json
              Time_Key    time
              Time_Format %Y-%m-%dT%H:%M:%S.%L
      [PARSER]
              Name        wso2
              Format      regex
              Regex       \[(?<date>\d{2,4}\-\d{2,4}\-\d{2,4} \d{2,4}\:\d{2,4}\:\d{2,4}\,\d{1,6})\]  (?<log_level>[^\s]+) \{(?<class>[\s\S]*)\} ([-]) (?<service>\{[\s\S]*\})?(?<message>.*)
              Time_Key    date
                  Time_Format %Y-%m-%d %H:%M:%S,%L
    • fluentBit.conf file

      [SERVICE]
          Flush        1
          Daemon       Off
          Log_Level    info
          Parsers_File <Location/parsers.conf>
      
      [INPUT]
          Name tail
          Path <MI_HOME>/repository/logs/*.log
          Mem_Buf_Limit  500MB
          Parser wso2
      
      [OUTPUT]
          Name loki
          Match *
          Url http://localhost:3100/loki/api/v1/push
          BatchWait 1
          BatchSize 30720
          Labels {job="fluent-bit"}
          LineFormat json
          LabelMapPath <Location/labelmap.json>
  4. Follow the instructions below to build the Fluent Bit output plugin before starting Fluent Bit:

    1. Clone the grafana/loki git repository.
    2. To build the Fluent Bit plugin, execute the following command.

      make fluent-bit-plugin

      For more details, see Fluent Bit Output Plugin readme file.

    3. Copy and save the path of the out_loki.so file.

  5. Open a new terminal and navigate to the <FluentBit_Home> directory.

  6. Execute the following command:

    Tip

    Replace <location of out_loki.so file> with the path that you copied and saved in the previous step.

    fluent-bit -e <location of out_loki.so file> -c <fluentbit.conf file path>

    When Fluent Bit is successfully installed, you will see a log message.

Step 2.2 - Set up the Loki server

Grafana Loki aggregates and processes the logs from Fluent Bit.

Follow the instructions below to set up Grafana Loki:

  1. Download Loki v1.6.1 from the grafana/loki Git repository.

    Tip

    Be sure to select the appropriate OS version before downloading.

  2. Create a configuration file named loki-local-config.yaml for Loki, similar to the sample given below, and save it in a preferred location.

    Tip

    • You can use a text editor of your choice for this purpose.
    • You can change the given parameter values based on your requirement.
    auth_enabled: false
    
    server:
      http_listen_port: 3100
    
    ingester:
      lifecycler:
        address: 127.0.0.1
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
        final_sleep: 0s
      chunk_idle_period: 5m
      chunk_retain_period: 30s
      max_transfer_retries: 0
    
    schema_config:
      configs:
        - from: 2018-04-15
          store: boltdb
          object_store: filesystem
          schema: v11
          index:
            prefix: index_
            period: 168h
    
    storage_config:
      boltdb:
        directory: /tmp/loki/index
    
      filesystem:
        directory: /tmp/loki/chunks
    
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    
    chunk_store_config:
      max_look_back_period: 0s
    
    table_manager:
      retention_deletes_enabled: false
      retention_period: 0s
  3. Unzip the file you downloaded in step 1.

    The directory that is created as a result is referred to as <GrafanaLoki_Home> from hereon.

  4. Open a new terminal and navigate to <GrafanaLoki_Home>.

  5. Execute the following command:

    ./loki-darwin-amd64 -config.file=./loki-local-config.yaml

Step 2.3 - Configure Grafana to visualize logs

Follow the instructions below to add Loki as a datasource in Grafana:

You need to do this in order to configure Grafana to display logs.

  1. Start you Grafana server.

    Tip

    The procedure to start Grafana depends on your operating system and the installation process. For example, If your operating system is Mac OS and you have installed Grafana via Homebrew, you start Grafana by issuing the brew services start grafana command.

  2. Access Grafana via http://localhost:3000/.

  3. In the Data Sources section, click Add your first data source. In the Add data source page that appears, click Select for Loki.

    Select Loki as Data Source

  4. In the Add data source page -> Settings tab, update the configurations for Loki.

  5. Click Save & Test.

    If the datasource is successfully configured, it is indicated via a message.

Step 3 - Optionally, integrate the Message Tracing add-on

Once you have successfully set up the minimum deployment, you need to set up the message tracing add-on using Jaeger.

Note

It is not essential to set up minimum deployment to observe the traces through Jaeger. You can do this without the minimum deployment as well.

Step 3.1 - Set up Jaeger

Download and install Jaeger.

Note

  • There are some limitations in the Jaeger client, which by default uses a UDP sender as mentioned in the Jaeger documentation. If the payload size exceeds 65 KB, spans might get lost in the Jaeger console.
  • Jaeger sampler types can also play a major role in tracing. Depending on the TPS, the sampler type should be carefully chosen.
  • In general, before including tracing in production deployments, it is essential to look into performance tests and scaling requirements. For details on how to achieve better performance, see the Jaeger performance tuning guide.

Step 3.2 - Set up the Micro Integrator

Follow the instructions below to configure the Micro Integrator to publish tracing information:

Add the following configurations to the deployment.toml file (stored in the <MI_HOME>/conf/).

```toml
[mediation]
flow.statistics.capture_all= true
stat.tracer.collect_payloads= true
stat.tracer.collect_mediation_properties= true

[opentracing]
enable = true
logs = true
manager_host = "localhost"
agent_host = "localhost"
```

Note

The service name used to initialize the JaegerTracer can be configured using the environment variable SERVICE_NAME as shown below.

export SERVICE_NAME=customServiceName
SERVICE_NAME is set to WSO2-SYNAPSE by default.

Step 3.3 - Configure Grafana to visualize tracing data

In order to configure Grafana to display tracing information, follow the steps given below.

  1. Add Jaeger as a datasource:

    1. Access Grafana via localhost:3000 and sign in.

    2. Click the Configuration icon in the left menu and then click Data Sources.

      Open Data sources

    3. Click Add data source to open the Add data source page where all the available data source types are displayed. Here, click Jaeger.

      Select Jaeger

      This opens the Data Sources/Jaeger dialog box.

    4. In the Data Sources/Jaeger dialog box, enter the URL of the Jaeger query component in the URL field in the http://host:port format as shown below.

      Enter Basic Jaeger Information

    5. Click Save and Test. If the data source is successfully configured, it is indicated via a message.

  2. Set up drill-down links to the Jaeger UI in service-level dashboards.

    1. Navigate to the settings section of the service-level dashboard by clicking the cogwheel icon in the upper-right corner.

    2. Click Variable. This opens the following view.

      Variables view

    3. Edit the JaegerHost variable and provide your Jaeger query component hostname and port in the host:port syntax as shown below.

      Constant options

    4. Click Save

    You need to perform the above steps for all the service-level dashboards (i.e., Proxy Service dashboard, API Service Dashboard, and Inbound Endpoint dashboard).

Once Grafana is successfully configured to visualize statistics, you should be correctly redirected to the Jaeger UI from the Response Time widget of each service-level dashboard as shown below.

jaeger ui

What's Next?

If you have successfully set up your analytics deployment, see the instructions on viewing cloud native observability statistics.

Top