Skip to content

Pattern 1: API-M Deployment with All-in-One HA Setup

This deployment consists of a highly available API-M cluster with multiple nodes of the API-M runtime. You can use this pattern if you expect to receive moderate traffic and require high availability in your environment.

active-active api-m deployment

Info

For advanced details on this deployment pattern, please refer to the official documentation.

Contents

Prerequisites

Before you begin, ensure you have the following prerequisites in place:

Step 1 - Set Up Basic Configurations

Info

The following tools and configurations are necessary for deploying WSO2 API-M in a Kubernetes environment.

  1. Install the required tools:
  2. Git
  3. Helm

  4. Kubernetes client

  5. Ensure you have a running Kubernetes cluster.

  6. Install the NGINX Ingress Controller.

  7. Add the WSO2 Helm chart repository: 

    helm repo add wso2 https://helm.wso2.com && helm repo update

Step 2 - Build Docker Images

  • WSO2 product Docker images are used for the Kubernetes deployment.

WSO2 product Docker images available at DockerHub package General Availability (GA) versions of WSO2 products with no WSO2 Updates.

For a production-grade deployment of the desired WSO2 product version, it is highly recommended to use the relevant Docker image that includes WSO2 Updates, available at the WSO2 Private Docker Registry. To use these images, you need an active WSO2 Subscription.

  • WSO2 API Manager 4.5.0 provides three Docker images:

  • All-in-one - wso2am

  • Since the products need to connect to databases at runtime, you must include the relevant JDBC drivers in the distribution. This can be done during the Docker image build stage. For example, you can add the MySQL driver as follows: 

    ADD --chown=wso2carbon:wso2 https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar ${WSO2_SERVER_HOME}/repository/components/lib

  • If there are any customizations to the JARs in the product, these can also be included in the Docker image itself rather than mounting them from the deployment level (assuming they are common to all environments).
  • Below is a sample Dockerfile to build a custom WSO2 APIM image. Depending on your requirements, you may refer to the following and make the necessary additions. The script will:
  • Use WSO2 APIM 4.5.0 as the base image

  • Copy third-party libraries to the <APIM_HOME>/lib directory

  • Dockerfile for All-in-One: 

    FROM docker.wso2.com/wso2am:4.5.0.0

ARG USER=wso2carbon
ARG USER_HOME=/home/${USER}
ARG WSO2_SERVER_NAME=wso2am
ARG WSO2_SERVER_VERSION=4.5.0
ARG WSO2_SERVER=${WSO2_SERVER_NAME}-${WSO2_SERVER_VERSION}
ARG WSO2_SERVER_HOME=${USER_HOME}/${WSO2_SERVER}

# Copy JDBC MySQL driver
ADD --chown=wso2carbon:wso2 https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar ${WSO2_SERVER_HOME}/repository/components/lib

  • Once the required changes have been made to the Dockerfile, you can use the following command to build the custom image. Replace CONTAINER_REGISTRY, IMAGE_REPO, and TAG accordingly: 

    docker build -t CONTAINER_REGISTRY/IMAGE_REPO:TAG .

  • After building your custom Docker image, push it to your container registry so it can be accessed by your Kubernetes cluster: 
    docker push CONTAINER_REGISTRY/IMAGE_REPO:TAG

Step 3 - Configure Database

  • Before running the API Manager, you must configure the databases and populate them with the initial data. All required database scripts are available in the dbscripts directory of the product pack. Locate the appropriate scripts for your chosen database engine and execute them accordingly. It is recommended to use two separate database users with limited permissions for enhanced security.

  • An example for MySQL is provided below: 

    CREATE DATABASE apim_db CHARACTER SET latin1;
CREATE DATABASE shared_db CHARACTER SET latin1;

GRANT ALL ON apim_db.* TO 'apimadmin'@'%';

CREATE USER 'sharedadmin'@'%' IDENTIFIED BY 'sharedadmin';
GRANT ALL ON shared_db.* TO 'sharedadmin'@'%';

    mysql -h <DB_HOST> -P 3306 -u sharedadmin -p -Dshared_db < './dbscripts/mysql.sql'
mysql -h <DB_HOST> -P 3306 -u apimadmin -p -Dapim_db < './dbscripts/apimgt/mysql.sql'

Minimal Configuration

If you want to quickly try out WSO2 API Manager on Kubernetes with minimal configuration, you can use the default values provided in the pre-configured YAML files.

Quick Start Configuration

This minimal configuration includes:

  • External database connection (requires setup)
  • Default keystore and truststore
  • Basic settings for a highly available deployment

Note: This deployment requires separate databases. Follow the steps in Step 2 - Build Docker Images to build the Docker images with JDBC drivers, and Step 3 - Configure Database to set up the database.

Deploy API Manager with minimal configuration using the following commands:

# Deploy first instance
helm install apim wso2/wso2am-all-in-one --version 4.5.0-3 -f https://raw.githubusercontent.com/wso2/helm-apim/main/docs/am-pattern-1-all-in-one-HA/default_values.yaml

Important

Naming conventions are important. If you want to change them, ensure consistency throughout your configuration.

Once the service is up and running, make sure you have the NGINX Ingress Controller deployed by following the steps outlined in the Add Ingress Controller section.

  • Once the service is up and running, deploy the NGINX Ingress Controller by following the steps outlined in 1.1 Add Ingress Controller.

Configuration

1. General Configuration of Helm Charts

The Helm charts for the API Manager deployment are available in the WSO2 Helm Chart Repository. You can either use the charts from the repository or clone the repository and use the charts from the local copy. - The Helm naming convention for APIM follows a simple pattern. The following format is used for naming the resources: <RELEASE_NAME>-<CHART_NAME>-<RESOURCE_NAME>

1.1 Add Ingress Controller

It is recommended to use the NGINX Ingress Controller suitable for your cloud environment or local deployment. Some sample annotations that could be used with the ingress resources are as follows:

  • The ingress class should be set to nginx in the ingress resource if you are using the NGINX Ingress Controller.

  • The following are some of the recommended annotations to include in the Helm charts for ingresses. These may vary depending on your requirements. Please refer to the documentation for more information about the annotations.

    ingressClass: "nginx"
ingress:
  tlsSecret: ""
  ratelimit:
    enabled: false
    zoneName: ""
    burstLimit: ""
  controlPlane:
    hostname: "am.wso2.com"
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      nginx.ingress.kubernetes.io/affinity: "cookie"
      nginx.ingress.kubernetes.io/session-cookie-name: "route"
      nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
    - You need to create a Kubernetes secret including the certificate and the private key, and include the name of the secret in the Helm charts. This will be used for TLS termination at the load balancer level by the ingress controller. Please refer to the documentation for more information. 
    kubectl create secret tls my-tls-secret --key <private key filename> --cert <certificate filename>

1.2 Mount Keystore and Truststore

  • If you are not including the keystore and truststore in the Docker image, you can mount them using a Kubernetes secret. The following steps show how to mount the keystore and truststore using a Kubernetes secret.
  • Create a Kubernetes secret with the keystore and truststore files. The secret should contain the primary keystore file, secondary keystore file, internal keystore file, and the truststore file. Note that the secret should be created in the same namespace in which you will be setting up the deployment.
  • Make sure to use the same secret name when creating the secret and when configuring the Helm chart.
  • If you are using a different keystore file name and alias, make sure to update the Helm chart configurations accordingly.
  • In addition to the primary, internal keystores, and truststore files, you can also include the keystores for HTTPS transport as well.
  • Refer to the following sample command to create the secret and use it in the APIM:
kubectl create secret generic apim-keystore-secret --from-file=wso2carbon.jks --from-file=client-truststore.jks --from-file=wso2internal.jks -n <namespace>

By default, this deployment uses the default keystores and truststores provided by the relevant WSO2 product. For advanced details regarding managing custom Java keystores and truststores in a container-based WSO2 product deployment, please refer to the official WSO2 container guide.

1.3 Encrypting Secrets

  • If you need to use the cipher tool to encrypt the passwords in the secret, first encrypt the passwords using the cipher tool. The cipher tool can be found in the bin directory of the product pack. The following command can be used to encrypt the password: 
    sh cipher-tool.sh -Dconfigure
  • The apictl can also be used to encrypt passwords. Reference can be found in the documentation.
  • Then, the encrypted values should be filled in the relevant fields of values.yaml.
  • Since the internal keystore password is required to resolve the encrypted value at runtime, you need to store the value in the cloud provider's secret manager. You can use the cloud provider's secret store to store the password of the internal keystore. The following section can be used to add the cloud provider's credentials to fetch the internal keystore password. Configuration for AWS can be as below: 
    internalKeystorePassword:
  # -- AWS Secrets Manager secret name
  secretName: ""
  # -- AWS Secrets Manager secret key
  secretKey: ""

    Please note that currently AWS, Azure, and GCP Secrets Managers are only supported for this.

1.4 Configure Docker Image and Databases

  • Add the following configurations to reflect the Docker image created previously in the Helm chart:

    wso2:
  deployment:
    image:
      imagePullSecrets:
        enabled: false
        username: ""
        password: ""        
      registry: ""
      repository: ""
      digest: ""

Info

If you are using a private Docker registry, you must enable imagePullSecrets.enabled and provide the username and password.

  • Provide the database configurations under the following section:

    wso2:
  apim:
    configurations:
      databases:
        apim_db:
          url: ""
          username: ""
          password: ""
        shared_db:
          url: ""
          username: ""
          password: ""
    - If you need to change the hostnames, update them under the Kubernetes ingress section. - Update the keystore passwords in the security section of the values.yaml file. - Review the descriptions of other configurations and modify them as needed to meet your requirements. A simple deployment can be achieved using the basic configurations provided in the values.yaml file. All configuration options for each Helm chart are documented in their respective component guides: - All-in-One - Universal Gateway - Update the admin credentials in the configuration directory: 
      # -- Super admin username
  adminUsername: ""
  # -- Super admin password
  adminPassword: ""

1.5 Configure SSL in Service Exposure

For WSO2 recommended best practices in configuring SSL when exposing the internal product services outside of the Kubernetes cluster, please refer to the official WSO2 container guide.

2. All-in-One Configurations

2.1 Configure Multiple Gateways

If you need to distribute the Gateway load, you can configure multiple API Gateway environments in WSO2 API Manager to publish to a single Developer Portal. See more... 

    gateway:
        # -- APIM Gateway environments
        environments:
        - name: "Default"
          type: "hybrid"
          gatewayType: "Regular"
          provider: "wso2"
          visibility:
          displayInApiConsole: true
          description: "This is a hybrid gateway that handles both production and sandbox token traffic."
          showAsTokenEndpointUrl: true
          serviceName: "apim-gw-wso2am-gateway-service"
          servicePort: 9443
          wsHostname: "websocket.wso2.com"
          httpHostname: "gw.wso2.com"
          websubHostname: "websub.wso2.com"
        - name: "Default_apk"
          type: "hybrid"
          provider: "wso2"
          gatewayType: "APK"
          displayInApiConsole: true
          description: "This is a hybrid gateway that handles both production and sandbox token traffic."
          showAsTokenEndpointUrl: true
          serviceName: "apim-gw-wso2am-gateway-service"
          servicePort: 9443
          wsHostname: "websocket.wso2.com"
          httpHostname: "default.gw.wso2.com:9095"
          websubHostname: "websub.wso2.com"

2.2 Configure User Store Properties

You can configure user store properties as described in this documentation:

    userStore:
    # -- User store type.
    type: "database_unique_id"
    # -- User store properties
    properties:
        ReadGroups: true

Warning

If you do not want to configure any of the above properties, you must remove the properties block from the YAML file.

2.4 Configure JWKS URL

By default, for the super tenant, the Resident Key Manager's JWKS URL is set to https://<HOSTNAME>:9443/oauth2/jwks. If you are using a virtual host like am.wso2.com that is not globally routable, this URL will be incorrect. You can configure the correct JWKS URL for the super tenant using the Helm chart as shown below:

wso2:
  apim:
    configurations:
      oauth_config:
        oauth2JWKSUrl: "https://localhost:9443/oauth2/jwks"

Tip

Use Key Manager's service name instead of localhost if you are using a different hostname for the Key Manager.

2.5 Deploy All-in-One

Now deploy the Helm chart using the following command after creating a namespace for the deployment. Replace <release-name> and <namespace> with appropriate values. Replace <helm-chart-path> with the path to the Helm deployment.

kubectl create namespace <namespace>
helm install <release-name> <helm-chart-path> --version 4.5.0-3 --namespace <namespace> --dependency-update -f values.yaml --create-namespace

2.6 Enable High Availability

To enable high availability, you can scale the deployment by increasing the number of replicas for the API Manager runtime. This can be done by modifying the highAvailability in the values.yaml file:

wso2:
  deployment:
    highAvailability: true

3. Add a DNS Record Mapping the Hostnames and the External IP

Obtain the external IP (EXTERNAL-IP) of the API Manager Ingress resources by listing the Kubernetes Ingresses: 

kubectl get ing -n <NAMESPACE>

If the defined hostnames (in the previous step) are backed by a DNS service, add a DNS record mapping the hostnames and the external IP (EXTERNAL-IP) in the relevant DNS service.

If the defined hostnames are not backed by a DNS service, for evaluation purposes you may add an entry mapping the hostnames and the external IP in the /etc/hosts file on the client side.

<EXTERNAL-IP> <kubernetes.ingress.management.hostname> <kubernetes.ingress.gateway.hostname> <kubernetes.ingress.websub.hostname> <kubernetes.ingress.websocket.hostname>

4. Access Management Consoles

  • API Manager Publisher: https://<kubernetes.ingress.management.hostname>/publisher
  • API Manager DevPortal: https://<kubernetes.ingress.management.hostname>/devportal
  • API Manager Carbon Console: https://<kubernetes.ingress.management.hostname>/carbon
  • Universal Gateway: https://<kubernetes.ingress.gateway.hostname>