Configuring Database and File System State Persistence

This section explains how to prevent the loss of data that can result from a system failure by persisting the state of WSO2 Streaming Integrator (WSO2 SI) periodically either into a database system or into the file system.

Prerequisites

Before configuring RDBMS database persistence, the following prerequisites must be completed.

  • One or more Siddhi applications must be running in the WSO2 SI server.

  • A working RDBMS instance that can be used for data persistence must exist.

  • The requirements of the datasource must be already defined.

  • Database persistence involves updating the databases connected to WSO2 Streaming Integrator with the latest information relating to the events that are being processed by WSO2 SI at a given time.

Configuring database system persistence

The supported databases are H2, MySQL, Postgres, MSSQL, and Oracle. The relevant JDBC driver JAR should be downloaded and added to the <SI_HOME>/lib directory to prior to using database system persistence.

To configure periodic data persistence, update the <SI_HOME>/conf/server/deployment.yaml file under state.persistence as follows:

Parameter Purpose Required Value
enabled This enables data persistence. true
intervalInMin The time interval in minutes that defines the interval in which state of Siddhi applications should be persisted. 1
revisionsToKeep The number of revisions to keep in the system. When a new persist takes place, the old revisions are removed. 3
persistenceStore The persistence store. org.wso2.carbon.stream.processor.core.persistence.DBPersistenceStore
config > datasource The datasource to be used in persisting the state. The provided datasource should be properly defined in the deployment.yaml. For detailed instructions of how to configure a datasource, see Configuring Datasources.
WSO2_PERSISTENCE_DB (Datasource with this name should be defined in wso2.datasources)
config > table The table that should be created and used for the persisting of the state.
PERSISTENCE_TABLE

The following is a sample segment of the required configurations in the <SI_HOME>/conf/server/deployment.yaml file to configure file system persistence.

Sample deployment.yaml segment

  state.persistence:
    enabled: true
    intervalInMin: 1
    revisionsToKeep: 3
    persistenceStore: org.wso2.carbon.stream.processor.core.persistence.DBPersistenceStore
    config:
      datasource: <DATASOURCE NAME>   # A datasource with this name should be defined in wso2.datasources namespace
      table: <TABLE NAME>

Configuring file system persistence

This section explains how to persist the states of Siddhi applications during a required time interval in the file system in order to maintain back-ups. To configure state persistence, update the <SI_HOME>/conf/server/deployment.yaml file under state.persistence as follows:

Parameter Purpose Required Value
enabled This enables data persistence. true
intervalInMin The time interval in minutes that defines the interval in which state of Siddhi applications should be persisted
1
revisionsToKeep The number of revisions to keep in the system. When a new persist takes place, the old revisions are removed. 3
persistenceStore The persistence store.
org.wso2.carbon.stream.processor.core.persistence.FileSystemPersistenceStore
config > location A fully qualified folder location to where the revision files should be persisted.
siddhi-app-persistence

The following is a sample segment of the required configurations in the <SI_HOME>/conf/server/deployment.yaml file to configure file system persistence.

Sample deployment.yaml segment

state.persistence:
enabled: true
intervalInMin: 1
revisionsToKeep: 2
persistenceStore: org.wso2.carbon.stream.processor.core.persistence.FileSystemPersistenceStore
config:
    location: siddhi-app-persistence
Top