Using Failover Endpoints¶
Example 1: Failover with one address endpoint¶
When message failure is not tolerable even though there is only one service endpoint, then failovers are possible with a single endpoint as shown in the below configuration.
Synapse configuration¶
Following is a sample REST API configuration that we can used to implement this scenario. See the instructions on how to build and run this example.
<api xmlns="http://ws.apache.org/ns/synapse" name="TestAPI" context="/test">
<resource methods="GET">
<inSequence>
<call>
<endpoint name="SampleFailover">
<failover>
<endpoint name="Sample_First" statistics="enable" >
<address uri="http://localhost:123/myendpoint" statistics="enable" trace="disable">
<timeout>
<duration>60000</duration>
</timeout>
<markForSuspension>
<errorCodes>101504, 101505, 101500</errorCodes>
<retriesBeforeSuspension>3</retriesBeforeSuspension>
<retryDelay>10</retryDelay>
</markForSuspension>
<suspendOnFailure>
<initialDuration>1000</initialDuration>
<progressionFactor>2</progressionFactor>
<maximumDuration>64000</maximumDuration>
</suspendOnFailure>
</address>
</endpoint>
</failover>
</endpoint>
</call>
<respond/>
</inSequence>
</resource>
</api>
In the above example, the Sample_First
endpoint is
marked as Timeout
if a connection times out, closes,
or sends IO errors after retrying for 60000
milliseconds.
When one of the errors of the specified codes occur (i.e.,
101504, 101505
and 101500)
, the
failover will retry using the first non-suspended endpoint. In this
case, it is the same endpoint ( Sample_First
). It
will retry until the retry count (i.e. 3 in the above example) becomes 0
with a delay as specified by the <retryDelay>
property (i.e., 10
milliseconds in the above example).
For all the other errors, it will be marked as
Suspended
. For more information about these states
and properties, see Endpoint Error Handling.
Info
The retry count is per endpoint, not per message. The retry happens in parallel. Since messages come to this endpoint via many threads, the same message may not be retried three times. Another message may fail and can reduce the retry count.
In this configuration, we assume that these errors are rare and if they happen once in a while, it is okay to retry again. If they happen frequently and continuously, it means that it requires immediate attention to get it back to normal state.
Build and run (example 1)¶
Create the artifacts:
- Set up WSO2 Integration Studio.
- Create an integration project with an ESB Configs module and an Composite Exporter.
- Create the REST API with the configurations given above.
- Deploy the artifacts in your Micro Integrator.
Send the following request to invoke the API:
curl -v -X GET "http://localhost:8290/test"
The following logs are printed if the endpoint is unreachable:
[2019-10-29 12:10:52,557] WARN {API_LOGGER.TestAPI} - ERROR_CODE : 101503 ERROR_MESSAGE : Error connecting to the back end
[2019-10-29 12:10:52,558] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : Sample_First with address http://localhost/myendpoint will be marked SUSPENDED as it failed
Example 2: Failover with multiple address endpoints¶
When a message reaches a failover endpoint with multiple address
endpoints, it will go through its list of endpoints to pick the first
one in Active
or Timeout
state
(not in the Suspended
state). Then, it will send the
message using that particular endpoint.
If a failure occurs with the first endpoint within the failover group
and if this error does not put the first endpoint into
Suspended
state, the retry will happen using the same
endpoint.
However, if the first endpoint is suspended or if an error occurs while
sending the message with the first endpoint, the failover endpoint will
go through the endpoint list again from the beginning and will try to
send the requests using the next endpoint, which is in the
Active
or Timeout
state.
Nevertheless, when the first endpoint becomes ready to send again, it
will try again on the first endpoint, even though the second endpoint is
still active. For more information about these states and properties,
see Endpoint Error Handling.
Synapse configuration¶
Following is a sample REST API configuration that we can used to implement this scenario. See the instructions on how to build and run this example.
Multiple address endpoints are used in this example.
<api xmlns="http://ws.apache.org/ns/synapse" name="TestAPI" context="/test">
<resource methods="GET">
<inSequence>
<call>
<endpoint>
<failover>
<endpoint name="fooEP">
<http uri-template="http://localhost:8080/foo">
<timeout>
<duration>10000</duration>
<responseAction>fault</responseAction>
</timeout>
<suspendOnFailure>
<errorCodes>101503,101504,101505,101507</errorCodes>
<initialDuration>100</initialDuration>
<progressionFactor>1.0</progressionFactor>
<maximumDuration>30000</maximumDuration>
</suspendOnFailure>
</http>
</endpoint>
<endpoint name="barEP">
<http uri-template="http://localhost:8080/bar">
<timeout>
<duration>10000</duration>
<responseAction>fault</responseAction>
</timeout>
<suspendOnFailure>
<errorCodes>101503,101504,101505,101507</errorCodes>
<initialDuration>100</initialDuration>
<progressionFactor>1.0</progressionFactor>
<maximumDuration>30000</maximumDuration>
</suspendOnFailure>
<retryConfig>
<disabledErrorCodes>101507,101503</disabledErrorCodes>
</retryConfig>
</http>
</endpoint>
</failover>
</endpoint>
</call>
<respond/>
</inSequence>
</resource>
</api>
Note
The <retryConfig>
property configures the last child endpoint to stop retying by ending the loop (i.e. to make the endpoint respond back to the service), after attempting to send requests to all the child endpoints and when all the attempts fail.
<retryConfig>
<disabledErrorCodes>101507,101504</disabledErrorCodes>
</retryConfig>
According to this configuration, the following error codes are used for sending the endpoint into Timeout
state: 101504
and 101507
.
Build and run (example 2)¶
Create the artifacts:
- Set up WSO2 Integration Studio.
- Create an ESB Solution project.
- Create the REST API with the configurations given above.
- Deploy the artifacts in your Micro Integrator.
Invoke the sample API by executing the following command:
curl -v -X GET "http://localhost:8290/test"
The following logs are getting printed if the endpoint is unreachable:
[2019-10-29 13:38:24,354] WARN {API_LOGGER.TestAPI} - ERROR_CODE : 101503 ERROR_MESSAGE : Error connecting to the back end
[2019-10-29 13:38:24,354] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : fooEP with address http://localhost:8080/foo will be marked SUSPENDED as it failed
[2019-10-29 13:38:24,354] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : fooEP with address http://localhost:8080/foo - current suspend duration is : 100ms - Next retry after : Tue Oct 29 13:38:24 IST 2019
[2019-10-29 13:38:24,355] WARN {org.apache.synapse.endpoints.FailoverEndpoint} - AnonymousEndpoint Detect a Failure in a child endpoint : Endpoint [fooEP]
[2019-10-29 13:38:24,356] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : localhost/127.0.0.1:8080
[2019-10-29 13:38:24,357] WARN {API_LOGGER.TestAPI} - ERROR_CODE : 101503 ERROR_MESSAGE : Error connecting to the back end
[2019-10-29 13:38:24,357] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : barEP with address http://localhost:8080/bar will be marked SUSPENDED as it failed
[2019-10-29 13:38:24,357] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : barEP with address http://localhost:8080/bar - current suspend duration is : 100ms - Next retry after : Tue Oct 29 13:38:24 IST 2019
Top