Production Deployment Guideline¶
Kubernetes should be used for Choreo Connect production deployments. The common guidelines for deploying Choreo Connect in a production environment using the Helm deployment option provided below.
You can deploy Choreo Connect using one of the following options.
This document walk you through following sections.
- Common Configurations - Common for both Standalone Gateway and API Manager as Control Plane
- Mode 1: API Manager as Control Plane Configurations
- Mode 2: Standalone Gateway Configurations
Common Configurations¶
Configuring Keystore¶
By default, Choreo Connect ships with default keystore certificates, and it is recommended to replace them with Self-Signed or CA signed certificates. Following steps describes steps to generate self-signed certificates and configure keystore of Choreo Connect Adapter, Enforcer and Router. If you have CA signed certificates you can skip Step: 1 of the following.
Step 1: Generate self-signed certs in PEM format (Optional)¶
Generate Self-Signed certs in PEM format by executing following sample commands. Let's create a directory called certs
to hold certificates for Adapter, Enforcer and Router.
mkdir -p certs/adapter certs/enforcer certs/router
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
-subj "/C=<COUNTRY_CODE>/ST=<STATE_NAME>/L=<LOCALITY_NAME>/O=<ORGANIZATION_NAME>/OU=<ORGANIZATION_UNIT_NAME>/CN=<COMMON_NAME>"\
-extensions SAN \
-config <(cat /etc/ssl/openssl.cnf \
<(printf "\n[SAN]\nsubjectAltName=DNS:<DNS1>,DNS:<DNS2>")) \
-keyout <KEY_FILE_PATH> \
-out <CERT_FILE_PATH>
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
-subj "/C=US/ST=CA/L=Mountain View/O=WSO2, Inc./OU=Choreo Connect/CN=yourdomain.com" \
-extensions SAN \
-config <(cat /etc/ssl/openssl.cnf \
<(printf "\n[SAN]\nsubjectAltName=DNS:adapter")) \
-keyout certs/adapter/tls.key \
-out certs/adapter/tls.crt
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
-subj "/C=US/ST=CA/L=Mountain View/O=WSO2, Inc./OU=Choreo Connect/CN=yourdomain.com" \
-extensions SAN \
-config <(cat /etc/ssl/openssl.cnf \
<(printf "\n[SAN]\nsubjectAltName=DNS:enforcer")) \
-keyout certs/enforcer/tls.key \
-out certs/enforcer/tls.crt
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
-subj "/C=US/ST=CA/L=Mountain View/O=WSO2, Inc./OU=Choreo Connect/CN=yourdomain.com" \
-extensions SAN \
-config <(cat /etc/ssl/openssl.cnf \
<(printf "\n[SAN]\nsubjectAltName=DNS:router")) \
-keyout certs/router/tls.key \
-out certs/router/tls.crt
By default, SSL Hostnames for adapter and enforcer is adapter
and enforcer
, in case if you want to have your own SAN names in the certificates, update the following values in the helm release with the SAN names in the certificates.
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
wso2.deployment.adapter.security.sslHostname=<ADAPTER_HOST_NAME>
wso2.deployment.gatewayRuntime.enforcer.security.sslHostname=<ENFORCER_HOST_NAME>
wso2.deployment.adapter.security.sslHostname=adapter
wso2.deployment.gatewayRuntime.enforcer.security.sslHostname=enforcer
Step 2: Create Kubernetes TLS Secrets¶
Create TLS secrets in the namespace that you wish to install Choreo Connect. Following are sample commands to create TLS secrets for above generated cert files.
Info
Change the value NAMESPACE_OF_CC=<NAMESPACE_OF_CC>
in following samples with the namespace that you wish to install Choreo Connect.
kubectl create secret tls <SECRET_NAME> -n <NAMESPACE_OF_CC> --cert=<CERT_PATH> --key=<KEY_PATH>
kubectl create secret tls adapter-keystore -n $NAMESPACE_OF_CC --cert=certs/adapter/tls.crt --key=certs/adapter/tls.key
kubectl create secret tls enforcer-keystore -n $NAMESPACE_OF_CC --cert=certs/enforcer/tls.crt --key=certs/enforcer/tls.key
kubectl create secret tls router-keystore -n $NAMESPACE_OF_CC --cert=certs/router/tls.crt --key=certs/router/tls.key
Step 3: Configure Secrets¶
You can set the keystore secrets in the same namespace that Choreo Connect is going to be installed. Set the following values.
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
keystore:
key:
secretName: <SECRET_NAME_1>
subPath: <KEY_OF_THE_SECRET_CONTAINS_THE_CERT>
cert:
secretName: <SECRET_NAME_2>
subPath: <KEY_OF_THE_SECRET_CONTAINS_THE_CERT>
wso2:
deployment:
adapter:
security:
keystore:
key:
secretName: "adapter-keystore"
subPath: "tls.key"
cert:
secretName: "adapter-keystore"
subPath: "tls.crt"
wso2:
deployment:
gatewayRuntime:
enforcer:
security:
keystore:
key:
secretName: "enforcer-keystore"
subPath: "tls.key"
cert:
secretName: "enforcer-keystore"
subPath: "tls.crt"
wso2:
deployment:
gatewayRuntime:
router:
security:
keystore:
key:
secretName: "router-keystore"
subPath: "tls.key"
cert:
secretName: "router-keystore"
subPath: "tls.crt"
When you update the Keystore of any component, the Truststore of other components get updated automatically, so no need to update Truststore manually.
Configuring Truststore¶
You can explicitly mount certs to the truststore of each component. The following is a sample how to define the truststore. If you have created a secret in the same namespace that Choreo Connect going to be installed, you can refer them in the config as follows.
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
...:
security:
truststore:
- secretName: <SECRET_NAME_1>
subPath: <KEY_OF_THE_SECRET_CONTAINS_THE_CERT>
mountAs: <FILE_NAME_OF_THE_MOUNTED_CERT>
- secretName: <SECRET_NAME_2>
subPath: <KEY_OF_THE_SECRET_CONTAINS_THE_CERT>
mountAs: <FILE_NAME_OF_THE_MOUNTED_CERT>
wso2:
deployment:
adapter:
security:
truststore:
- secretName: "abc-certs"
subPath: "tls.crt"
mountAs: "tls.crt"
wso2:
deployment:
gatewayRuntime:
enforcer:
security:
truststore:
- secretName: "controlplane-cert"
subPath: "wso2carbon.pem"
mountAs: "wso2carbon.pem"
- secretName: "abc-certs"
subPath: "tls.crt"
mountAs: "tls.crt"
Note
You can use mountAs
to have your own file name inside the container. For example, you can mount the cert tls.crt
in the secret abc-certs
as abc-tls-cert.crt
. Refer to the following example values.yaml.
wso2:
deployment:
adapter:
security:
truststore:
- secretName: "abc-certs"
subPath: "tls.crt"
mountAs: "abc-tls-cert.crt"
If mountAs
is not defined, the Helm chart will rename the file as <secretName>-<subPath replace '.' with '-' >.pem
.
You can verify the cert is successfully mounted to the container by executing the following command.
NAMESPACE=<NAMESPACE>
kubectl exec -n "$NAMESPACE" \
"$(kubectl get pod -n $NAMESPACE -l app.kubernetes.io/component=<COMPONENT_NAME> -o jsonpath='{.items[0].metadata.name}')" \
-c <CONTAINER_NAME> -- ls -alh /home/wso2/security/truststore/
NAMESPACE=<NAMESPACE>
kubectl exec -n "$NAMESPACE" \
"$(kubectl get pod -n $NAMESPACE -l app.kubernetes.io/component=choreo-connect-adapter -o jsonpath='{.items[0].metadata.name}')" \
-c choreo-connect-adapter -- ls -alh /home/wso2/security/truststore/
NAMESPACE=<NAMESPACE>
kubectl exec -n "$NAMESPACE" \
"$(kubectl get pod -n $NAMESPACE -l app.kubernetes.io/component=choreo-connect-gateway-runtime -o jsonpath='{.items[0].metadata.name}')" \
-c choreo-connect-enforcer -- ls -alh /home/wso2/security/truststore/
Change Default Passwords¶
You can change the default passwords or environment variables by setting/overriding them by defining them in the field envOverride
.
These passwords or environment variables can be set directly as a plain text or can be referenced from a Kubernetes Secret.
This configuration is the same way you can define environment variables in Kubernetes Pods.
In the following sample, the enforcer_admin_pwd
is set using the value
field and the tm_admin_pwd
is referred using a secret.
wso2:
deployment:
gatewayRuntime:
enforcer:
envOverride:
- name: enforcer_admin_pwd
value: admin
- name: tm_admin_pwd
valueFrom:
secretKeyRef:
name: my-secret
key: tm_password
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
Following are the default values, update to read them from a Kubernetes secret.
wso2:
deployment:
adapter:
envOverride:
- name: cp_admin_pwd
value: admin
- name: adapter_admin_pwd
value: admin
wso2:
deployment:
gatewayRuntime:
enforcer:
envOverride:
- name: apim_admin_pwd
value: "admin"
- name: enforcer_admin_pwd
value: "admin"
- name: tm_admin_pwd
value: "admin"
Configure Ingress Secrets¶
By default, the helm chart installation of Choreo Connect creates default certificates for ingress resources. Providing TLS Secret name in the following values will replace those default certs with provided certs.
Note
These Ingress Secrets should be TLS secrets
wso2.deployment.adapter.ingress.tlsSecretName=<TLS_CERT_SECRET_IN_THE_SAME_NAMESPACE>
wso2.deployment.gatewayRuntime.router.ingress.tlsSecretName=<TLS_CERT_SECRET_IN_THE_SAME_NAMESPACE>
Disable Test Token Issuer¶
Choreo Connect provides the ability to generate a JWT as a test key to access the APIs. It is recommended to disable the /testkey
endpoint in the Production environments.
Refer the document Generate a Test JWT to read more about Test JWT.
Set the following value to false
when installing the helm chart or set it in the values.yaml file.
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
wso2.deployment.gatewayRuntime.enforcer.security.testTokenIssuer.enabled=false
Create Custom Docker Image (Optionally)¶
You can create your own docker image using Choreo Connect docker images as the base image.
For example lets say you want to replace the CA certificates of Choreo Connect Router with your own set of certificates. Create a Dockerfile as follows.
FROM wso2/choreo-connect-router:1.1.0
<YOUR_DOCKER_COMMANDS>
FROM wso2/choreo-connect-router:1.1.0
COPY my-ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
Make sure the files you are adding or copying (with commands ADD
and COPY
) exist in the context path.
docker build -t <IMAGE_NAME> -f <DOCKER_FILE_PATH> <CONTEXT>
docker build -t myimages/choreo-connect-router:1.1.0 -f Dockerfile .
Mode 1: API Manager as Control Plane Configurations¶
The default deployment mode of the Choreo Connect is Standalone Mode. You can change it by specifying the value wso2.deployment.mode
as APIM_AS_CP
.
Then configure the externally installed WSO2 API Manager by updating the following values of the helm release.
wso2:
apim:
controlPlane:
# Hostname of the control plane
hostName: <HOST_NAME_OF_THE_CONTROL_PLANE>
# K8s service name (if in another namespace, `<serviceName>.<namespace>`) of the control plane
serviceName: <K8S_SERVICE_NAME>.<K8S_NAMESPACE_OF_THE_CONTROL_PLANE>
trafficManager:
# K8s service name of the traffic manager. If not defined, default to control plane service name
serviceName: <K8S_SERVICE_NAME>.<K8S_NAMESPACE_OF_THE_TRAFFIC_MANAGER>
deployment:
# Deployment option: one of "STANDALONE" or "APIM_AS_CP"
# Refer deployment options: https://apim.docs.wso2.com/en/4.1.0/deploy-and-publish/deploy-on-gateway/choreo-connect/getting-started/deploy/cc-deploy-overview/
mode: APIM_AS_CP
wso2:
apim:
controlPlane:
# Hostname of the control plane
hostName: am.wso2.com
# K8s service name (if in another namespace, `<serviceName>.<namespace>`) of the control plane
serviceName: wso2am-single-node-am-service.apim
trafficManager:
# K8s service name of the traffic manager. If not defined, default to control plane service name
serviceName: ""
deployment:
# Deployment option: one of "STANDALONE" or "APIM_AS_CP"
# Refer deployment options: https://apim.docs.wso2.com/en/4.1.0/deploy-and-publish/deploy-on-gateway/choreo-connect/getting-started/deploy/cc-deploy-overview/
mode: "APIM_AS_CP"
Please follow the document about Deploying Choreo Connect on Kubernetes With WSO2 API Manager as a Control Plane - Helm Artifacts for deploying Choreo Connect.
Mode 2: Standalone Gateway Configurations¶
Deploy APIs as Immutable Gateway¶
The API Controller apictl
can be used to deploy APIs in the standalone mode. Those APIs deployed with apictl
will be lost if the Adapter container restarts for any reason.
Hence, in a production deployment with the Standalone deployment option, it is recommend to create a custom docker image of Adapter by including the apictl projects
.
Step 1: Create Projects¶
Follow the steps (step 1 and step 2) in the document Deploy APIs as Immutable Gateway to create an apictl project
. You can include multiple apictl projects
.
For example lets create a directory apictl-projects-dir
and copy all apictl projects
(petstore
project for following sample) to this directory. These projects can be zip files, unzipped project directory or projects that are exported from WSO2 API Manager.
mkdir apictl-projects-dir
cp -r petstore/ apictl-projects-dir/petstore
Step 2: Create Custom Adapter Docker Image¶
Create a Dockerfile as follows.
FROM wso2/choreo-connect-adapter:1.1.0
COPY <DIR_WITH_APICTL_PROJECTS> /home/wso2/artifacts/apis
FROM wso2/choreo-connect-adapter:1.1.0
COPY apictl-projects-dir /home/wso2/artifacts/apis
Make sure the files you are adding or copying (with commands ADD
and COPY
) exist in the context path.
docker build -t <IMAGE_NAME> -f <DOCKER_FILE_PATH> <CONTEXT>
docker build -t myimages/choreo-connect-adapter-petstore:1.1.0 -f Dockerfile .
Step 3: Update Adapter Docker Image Name¶
Update the following values in the helm release with the Adapter docker image, image pull secrets. You can separate each gateway environments by specifying the value wso2.deployment.labelName
.
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
Important
Make sure to set wso2.deployment.adapter.apiArtifactsMountEmptyDir=false
. This field is available from Helm chart version 1.1.0.5
.
wso2:
deployment:
# Label (environment) name of the deployment
labelName: "<ENVIRONMENT_LABEL_NAME>"
# If a custom image must be used, define the docker registry. Default to DockerHub. If subscription specified it will be "docker.wso2.com"
dockerRegistry: "<DOCKER_REGISTRY>"
# Image pull secrets to pull images from docker registry. If subscriptions are specified a secret with subscriptions details are created and imagePullSecrets will be default to it.
imagePullSecrets: <LIST_OF_PULL_SECRETS>
adapter:
# Docker registry. If this value is not empty, this overrides the value in 'wso2.deployment.dockerRegistry'
dockerRegistry: <DOCKER_REGISTRY_FOR_ADAPTER>
# Image name for adapter
imageName: "<ADAPTER_IMAGE_NAME>"
# Image tag for adapter
imageTag: "<IMAGE_TAG>"
# Refer to the Kubernetes documentation on updating images (https://kubernetes.io/docs/concepts/containers/images/#updating-images)
imagePullPolicy: <IMAGE_PULL_POLICY>
# Mount an empty directory on API artifacts directory
apiArtifactsMountEmptyDir: false
wso2:
deployment:
# Label (environment) name of the deployment
labelName: "default"
# If a custom image must be used, define the docker registry. Default to DockerHub. If subscription specified it will be "docker.wso2.com"
dockerRegistry: ""
# Image pull secrets to pull images from docker registry. If subscriptions are specified a secret with subscriptions details are created and imagePullSecrets will be default to it.
imagePullSecrets: []
adapter:
# Docker registry. If this value is not empty, this overrides the value in 'wso2.deployment.dockerRegistry'
dockerRegistry: "myimages"
# Image name for adapter
imageName: "choreo-connect-adapter-petstore"
# Image tag for adapter
imageTag: "1.1.0"
# Refer to the Kubernetes documentation on updating images (https://kubernetes.io/docs/concepts/containers/images/#updating-images)
imagePullPolicy: IfNotPresent
# Mount an empty directory on API artifacts directory
apiArtifactsMountEmptyDir: false
Step 4: Disable the Adapter Rest API¶
Since we want to make our gateway immutable, we can disable the Adapter Rest API which is used by the apictl
communicate the Choreo Connect to
deploy, undeploy APIs. Update the following value of the helm release for that purpose.
Tip
You can find the complete values.yaml file with default values and configuration description in the kubernetes-microgateway GitHub repository.
wso2.deployment.adapter.security.adapterRestService.enabled="false"
Note
The above value is in String type which is one of "default"
, "true"
or "false"
. It is not a Boolean value.
Step 5: Deploy Choreo Connect¶
Please follow the document about Deploying Choreo Connect as a Standalone Gateway on Kubernetes - Helm Artifacts for deploying Choreo Connect.
Top