

Adding Debug and Trace Logs¶

Following explain how to add debug and trace logs for the main three components of the Choreo Connect which are Adapter, Enforcer and Router.

Adapter¶

Enable debug logs¶

Configure the logging in the log_config.toml in the directory <CHOREO-CONNECT_HOME>/docker-compose/<choreo-connect>/conf/.

The following log levels are supported at root and package level. "INFO", "WARN", "DEBG", "FATL", "ERRO", "PANC"

Additionally, log rotation can be configured as follows for adapter logs.

## Adapter root Level configurations

logfile = "logs/adapter.log" # This file will be created inside adapter container.
logLevel = "INFO"

[rotation]
MaxSize = 10 # In MegaBytes (MB)
MaxBackups = 3
MaxAge =  2   # In days
Compress = true

## Adapter package Level configurations

[[pkg]]
name = "github.com/wso2/adapter/internal/adapter"
logLevel = "INFO"

[[pkg]]
name = "github.com/wso2/adapter/internal/oasparser"
logLevel = "INFO"

Enforcer¶

Warning

Note that WSO2 API Manager 3.0.0, 3.1.0, 3.2.0, and 4.0.0 are affected by the Log4j2 zero-day vulnerability, which has been reported to WSO2 on 10^th December 2021. You can mitigate this vulnerability in your product by following our instructions and guidelines.

Enable debug logs¶

Configure log4j2.properties located in the directory <CHOREO-CONNECT_HOME>/docker-compose/<choreo-connect>/conf/. Make relevant packages to DEBUG level.

logger.enforcer.level = DEBUG

appender.ENFORCER_LOGFILE.fileName = logs/enforcer.log
appender.ENFORCER_LOGFILE.filePattern = /logs/enforcer-%d{MM-dd-yyyy}.log
appender.ENFORCER_LOGFILE.layout.type = PatternLayout
appender.ENFORCER_LOGFILE.layout.pattern = [%d][%X{traceId}] %5p - %m%ex%n
...

Enable access logs¶

Configure log4j2.properties located in the directory <CHOREO-CONNECT_HOME>/docker-compose/<choreo-connect>/conf/.

By making the value logger.mgw-enforcer-interceptors.level to DEBUG in the below configuration will enable access logs without restarting the enforcer.

logger.mgw-enforcer-interceptors.level = DEBUG

appender.MGW_ACCESS_LOG.fileName = logs/enforcer_access.log # This file will be created inside enforcer container.
appender.MGW_ACCESS_LOG.filePattern = /logs/enforcer_access-%d{MM-dd-yyyy}.log
appender.MGW_ACCESS_LOG.layout.type = PatternLayout
appender.MGW_ACCESS_LOG.layout.pattern = [%d] - %m%ex%n # Log pattern
...

The access log format will be as follow. It will print the server time, trace Id from envoy, gRPC service method, gRPC status code, response time according to the above configuration.

[2021-02-19 07:48:49,505] - 5920896249661898188 envoy.service.auth.v3.Authorization/Check 16 34
[2021-02-19 07:48:52,592] - 17895662172888229144 envoy.service.auth.v3.Authorization/Check 16 7

Router¶

Enable access logs.¶

Configure the following in the log_config.toml in the directory <CHOREO-CONNECT_HOME>/docker-compose/<choreo-connect>/conf/. Please follow command operators for more information on the supported options for log format, format configuration.

[accessLogs]
enable = false
logfile = "/tmp/envoy.access.log" # This file will be created inside router container.
format = "[%START_TIME%] '%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%' %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% '%REQ(X-FORWARDED-FOR)%' '%REQ(USER-AGENT)%' '%REQ(X-REQUEST-ID)%' '%REQ(:AUTHORITY)%' '%UPSTREAM_HOST%'\n"

Enable debug logs¶

Provide the log level as trailing arguments for the envoy command as follows.

Format

-l <level>, 
--log-level <level>
--component-log-level <component>:<level>,<component>:<level>...

Example

-l trace, 
--log-level trace
--component-log-level upstream:debug,connection:trace

For example, add the following line to the docker-compose.yaml file in the directory <CHOREO-CONNECT_HOME>/docker-compose/<choreo-connect>/.

  router:
    environment:
      - TRAILING_ARGS=--log-level trace

The following example configuration can log request headers and response headers, and ext_authz information.

  router:
    environment:
      - TRAILING_ARGS=--component-log-level http:debug,http2:debug,conn_handler:debug,ext_authz:trace

The following are different sections of an example log with the above log configuration. This log is generated by invoking an API request with /pet/findByStatus?status=sold path in Swagger Petstore API deployed to Choreo Connect. Different log sections are given in the order same as the order in which they are logged to the console.

Client to Router request headers

[2022-04-11 08:18:03.917][15][debug][http] [source/common/http/conn_manager_impl.cc:867] [C19][S3973774370179828478] request headers complete (end_stream=true):
':authority', 'localhost:9095'
':path', '/v2/pet/findByStatus?status=sold'
':method', 'GET'
'user-agent', 'curl/7.68.0'
'accept', 'application/json'
'authorization', 'Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1UWm1ZemsxWlRJeVkyTmlNR00wT1RJMFpqTXdNRE5rTldZek16QTJOVEl4TWpRelpERTFNdz09In0.eyJhdWQiOiJodHRwOlwvXC9vcmcud3NvMi5hcGltZ3RcL2dhdGV3YXkiLCJzdWIiOiJhZG1pbiIsInNjb3BlIjoicmVhZDpwZXRzIiwiaXNzIjoiaHR0cHM6XC9cL2xvY2FsaG9zdDo5MDk1XC90ZXN0a2V5Iiwia2V5dHlwZSI6IlBST0RVQ1RJT04iLCJleHAiOjE2NDk2Njg2NjUsImlhdCI6MTY0OTY2NTA2NSwianRpIjoiZDI3MTRiYmMtMWE5OC00ODc5LThmM2QtM2I5ZGYwNjI1NDcwIn0.MXkSAq98Zx32j5ijK4MLvlocMhjVW_xxW3ECeGVmHaRkqg43bO2PLtdoh952YBbK4xzK3mDhAxF1VxynDyD5u1qpu696e4w3UwF2pfU3hAdbG2fDdcm82KiSfydawOnivxFr-QX08tQxDTO9i5Fgo7nZMsZkNRN2er1asctDaDRcqcR00Jh6h1jyqBU6Ixf67dkZk61et8yNdGeSiuy-QXLVHsN8YbvXNyct1d1nVjZrE3d_kDkk_ZQlqHj327BXolCl_tiinj_bBDovmqIDNwTXkL47rtuWX6sB1uBJ0xBHtuZxT7-IgB1WUVvVcbAqNebQFs1bV8A-h0efQKKBdg'

Router to Enforcer request infomation

[2022-04-11 08:18:03.917][15][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:42] Sending CheckRequest: attributes {
  source {
    address {
      socket_address {
        address: "172.18.0.1"
        port_value: 40312
      }
    }
  }
  destination {
    address {
      socket_address {
        address: "172.18.0.5"
        port_value: 9095
      }
    }
    principal: "adapter"
  }
  request {
    time {
      seconds: 1649665083
      nanos: 917501000
    }
    http {
      id: "3973774370179828478"
      method: "GET"
      headers {
        key: ":authority"
        value: "localhost:9095"
      }
      headers {
        key: ":method"
        value: "GET"
      }
      headers {
        key: ":path"
        value: "/v2/pet/findByStatus?status=sold"
      }
      headers {
        key: ":scheme"
        value: "https"
      }
      headers {
        key: "accept"
        value: "application/json"
      }
     ...
     ...

Enforcer to Router response information

[2022-04-11 08:18:03.937][15][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:48] Received CheckResponse: status {
}
ok_response {
  headers {
    header {
      key: "X-JWT-Assertion"
      value: "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1UWm1ZemsxWlRJeVkyTmlNR00wT1RJMFpqTXdNRE5rTldZek16QTJOVEl4TWpRelpERTFNdz09In0=.eyJzdWIiOiJhZG1pbiIsImh0dHA6XC9cL3dzbzIub3JnXC9jbGFpbXNcL2FwcGxpY2F0aW9udGllciI6IlVubGltaXRlZCIsImh0dHA6XC9cL3dzbzIub3JnXC9jbGFpbXNcL3ZlcnNpb24iOiIxLjAuNiIsImh0dHA6XC9cL3dzbzIub3JnXC9jbGFpbXNcL2tleXR5cGUiOiJQUk9EVUNUSU9OIiwiaXNzIjoid3NvMi5vcmdcL3Byb2R1Y3RzXC9hbSIsImh0dHA6XC9cL3dzbzIub3JnXC9jbGFpbXNcL2FwcGxpY2F0aW9ubmFtZSI6ImFub246TUdXIiwia2V5dHlwZSI6IlBST0RVQ1RJT04iLCJodHRwOlwvXC93c28yLm9yZ1wvY2xhaW1zXC9lbmR1c2VyVGVuYW50SWQiOiIwIiwiaHR0cDpcL1wvd3NvMi5vcmdcL2NsYWltc1wvYXBwbGljYXRpb25VVUlkIjoiNWE2YmNiMGMtZjdjMi0zYmNkLWJiMGEtMDIxZWYyMWYyMWViIiwic2NvcGUiOiJyZWFkOnBldHMiLCJleHAiOjE2NDk2NjQwODEsImh0dHA6XC9cL3dzbzIub3JnXC9jbGFpbXNcL2FwcGxpY2F0aW9uaWQiOiI1YTZiY2IwYy1mN2MyLTNiY2QtYmIwYS0wMjFlZjIxZjIxZWIiLCJodHRwOlwvXC93c28yLm9yZ1wvY2xhaW1zXC91c2VydHlwZSI6IkFwcGxpY2F0aW9uX1VzZXIiLCJpYXQiOjE2NDk2NjUwODEsImp0aSI6Ijc1NWEwYzI4LWE2ODUtNDYzNC1hNzNiLWYzOGU4NDNmODYxNyIsImh0dHA6XC9cL3dzbzIub3JnXC9jbGFpbXNcL2FwaWNvbnRleHQiOiJcL3YyIn0=.iphgEpqjBVA48rGZdv14A_iLZE81V2p0iObUGg5UPefGHDVMZoi_w2Z30oMuBpRougXoPmMWkQ_Cj4llG31d9CPW7HkQyicHup4UH1n25L4Iv3D1Pny_NhDvDmeixT3PQ_FU-1NK1tB1lxW1c4WVoRfbrIqvD5ojYx6aPy7eBNp1Hx-pZib53rd_LKkmHXH0WGW_N-vjwhlXOlCu9pANkJeRx4QWDiNvooCEgwqG2Dwgh2z8ck5XWgCVLb4Pm24Zqg0GucKfy3LTBi7uupYvDIQtUfM_HExm2OimxyCwOYDpyRGMmRqGIS2zCRq5lt8Cc3HbPpa5rxJvCXF848Aj5A=="
    }
  }
  headers {
    header {
      key: "x-wso2-cluster-header"
      value: "carbon.super_clusterProd_localhost_SwaggerPetstore1.0.6"
    }
  }
  headers {
    header {
      key: "X-TRACE-KEY"
      value: "3973774370179828478"
    }
  }
  headers_to_remove: "api_key"
  headers_to_remove: "internal-key"
}
dynamic_metadata {
  fields {
    key: "keyType"
    value {
      string_value: "PRODUCTION"
    }
  }
  fields {
    key: "token"
    value {
      string_value: "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1UWm1ZemsxWlRJeVkyTmlNR00wT1RJMFpqTXdNRE5rTldZek16QTJOVEl4TWpRelpERTFNdz09In0.eyJhdWQiOiJodHRwOlwvXC9vcmcud3NvMi5hcGltZ3RcL2dhdGV3YXkiLCJzdWIiOiJhZG1pbiIsInNjb3BlIjoicmVhZDpwZXRzIiwiaXNzIjoiaHR0cHM6XC9cL2xvY2FsaG9zdDo5MDk1XC90ZXN0a2V5Iiwia2V5dHlwZSI6IlBST0RVQ1RJT04iLCJleHAiOjE2NDk2Njg2NjUsImlhdCI6MTY0OTY2NTA2NSwianRpIjoiZDI3MTRiYmMtMWE5OC00ODc5LThmM2QtM2I5ZGYwNjI1NDcwIn0.MXkSAq98Zx32j5ijK4MLvlocMhjVW_xxW3ECeGVmHaRkqg43bO2PLtdoh952YBbK4xzK3mDhAxF1VxynDyD5u1qpu696e4w3UwF2pfU3hAdbG2fDdcm82KiSfydawOnivxFr-QX08tQxDTO9i5Fgo7nZMsZkNRN2er1asctDaDRcqcR00Jh6h1jyqBU6Ixf67dkZk61et8yNdGeSiuy-QXLVHsN8YbvXNyct1d1nVjZrE3d_kDkk_ZQlqHj327BXolCl_tiinj_bBDovmqIDNwTXkL47rtuWX6sB1uBJ0xBHtuZxT7-IgB1WUVvVcbAqNebQFs1bV8A-h0efQKKBdg"
    }
  }
  fields {
    key: "tokenType"
    value {
      string_value: "JWT"
    }
  }
  ...
  ...

Router to client response headers

[2022-04-11 08:18:04.956][15][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C19][S3973774370179828478] encoding headers via codec (end_stream=false):
':status', '200'
'date', 'Mon, 11 Apr 2022 08:18:04 GMT'
'content-type', 'application/json'
'access-control-allow-origin', '*'
'access-control-allow-methods', 'GET, POST, DELETE, PUT'
'access-control-allow-headers', 'Content-Type, api_key, Authorization'
'server', 'envoy'
'x-envoy-decorator-operation', 'localhost:^/v2/pet/findByStatus(\?([^/]+))?$'

Follow Command line options for more information.

Enable Wire Logs¶

To enable wire logs, follow the instructions provided in Router Wire Logs.

Admin portal¶

The admin interface can be used to view statistics, envoy configurations, etc. For more information, please follow Envoy admin interface Configure host and port for the envoy admin interface and expose it.

Note

This endpoint is not protected with authentication, hence ensure to restrict access to this address in your network infrastructure.

For example, provide following to the docker-compose.yaml in the directory <CHOREO-CONNECT_HOME>/docker-compose/<choreo-connect>/.

  router:
    environment:
      - ROUTER_ADMIN_HOST=0.0.0.0
      - ROUTER_ADMIN_PORT=9000
    ports:
      - "9000:9000"

Top