

Building a CI/CD Pipeline for APIs Using WSO2 API Controller¶

In the modern digital-era, many organizations adopt digital transformation technologies to keep up with the changes. APIs are one of the basic building blocks used in a digitally-driven organization. When the number of APIs managed by the organization grows, they need to have an automated process to handle the rapid API development process. Having a proper continuous integration and continuous deployment (CI/CD) process would give an added advantage to your organization.

In this section, you will learn how to build an automated process using the WSO2 API Controller (apictl) with WSO2 API Manager (WSO2 API-M). You will see how a solution is built using a version control system (GitHub), artifact repository(JFrog Artifactory) and CI/CD Tool (Jenkins).

Setting up the environment¶

Step 1 - Setup Jenkins¶

Before you begin, you need to set up the Jenkins server on a remote machine that has public access. To use webhooks in GitHub and Artifactory repository to trigger Jenkins jobs, you need a publicly accessible Jenkins server URL. If you are trying this on a local machine, you may have to use a tool such as socketxp to create a tunnel.

In this section, the instructions are tested out on an Ubuntu server setup on https://cloud.google.com/compute .

Log in to the server and install the following.

Install jq using the command given below.
```
sudo apt-get install jq
```
Install the apictl tool. Extract to a location that is henceforth referred to as CTL_HOME.
Install Java.
Setup the Jenkins server following the instructions given here. You can choose to install the suggested default plugins. This will install the commonly-used plugins.
Register a webhook in the JForg Artifact repository to notify changes. To capture these events and trigger a Jenkins job, install a generic-webhook-trigger plugin.
Configure apictl home as an environment variable. Go to the Manage Jenkins section and select Configure System.

Under the Global properties select Environment variables and set the APICTL home as below and click Save.

Now the Jenkins server is set up.

Step 2 - Create GitHub repositories¶

In this section, two repositories are created to illustrate this scenario.

Source Repository -https://github.com/chamilaadhi/poc-cicd-source-repo
Deployment Repository - https://github.com/chamilaadhi/poc-cicd-deployment-repo

Source repository contains the files and metadata related to the API. Deployment repository contains the configurations related to each environment this APIs is getting deployed.

Step 2.1 - Setup source repository¶

This section shows how to trigger a Jenkins job when a change is made to the repository. Lets use GitHub webhooks to trigger a Jenkins job.

Go to the source repository and under the Settings, select Webhooks and add a webhook to the jenkins server.

Use your jenkins server URL (e.g., http://<public_ip>:8080/github-webhook/) as the URL and Content type as application/json.
Under the Which events would you like to trigger this webhook? section, select Let me select individual events, and select pull request and pushes.

Source repository is now configured.

Now let’s initialize the source repository. This will be done from the developer’s local machine.

Note

If you haven’t setup API Controller, set it up using the apictl tool

Clone the source repository. Navigate to the repository folder.

$ git clone https://github.com/chamilaadhi/poc-cicd-source-repo.git
$ cd poc-cicd-source-repo

Execute the following command.
```
$ apictl vcs init
```
This will create a vcs.yaml file in the repository. Commit it to the source repository.

Now the source repository is configured.

Step 2.2 - Setup deployment repository¶

Deployment repository will be used to gather configurations for each deployment. To separate out the configuration, we will use git branches. Create a branch named ‘dev’ to keep the configuration related to the dev environment.

Step 3 - Setup JFrog Artifactory¶

Before you begin

Create a free cloud repository using https://jfrog.com/start-free/#saas. After the registration is complete, Login to https://<name>.jfrog.io/ui/

Create a user to upload artifacts. Login using admin credentials and go to Identity and Access → Users and create a user. All roles are selected in this scenario.
Make the repo accessible public. Go to Security → Settings and select Allow Anonymous Access
Create a repository to upload the artifacts. Go to Repositories and select Add Repositories. Select Generic type and provide a name (e.g., myrepo).

After saving, You should be able to access https://<your_org>.jfrog.io/artifactory/<repo_name>/.

Try uploading a sample resource. An example is given below.
```
$ touch resource.txt
$ curl -u repouser:<password> -X PUT https://xxxx.jfrog.io/artifactory/myrepo/ -T resource.txt
```
Access the repository URL to see the resource.
Configure a webhook to triggerb the Jenkins job.

Create a webhook to trigger the Jenkins job when a new resource is uploaded to the repository. Go to General → Webhooks and create a new one.

URL - http://<jenkins_host>:8080/generic-webhook-trigger/invoke?token=123**

Add the Event set as Artifact was deployed.

Select the repository from the new window that appears.

Step 4 - Setup API-M instances¶

Download the WSO2 API Manager 4.1.0 here and start the instance. Set this in a separate instance as the dev instance.

Step 5 - Configure Jenkins Jobs¶

Step 5.1 - Setup Global variables¶

As shown in the Setup Jenkins Section, Go to Manage Jenkins section and select Configure System. Under the Global properties select Environment variables and set following variables with your environment related details.

Variable	Description
APIM_DEV_HOST	IP or host name of the dev instance (ex 10.100.5.96 )
APIM_PROD_HOST	IP or host name of the prod instance (ex 10.100.5.97 )
ARTIFACTORY_HOST	Artifactory host name (ex: testrepo.jfrog.io)
ARTIFACTORY_PWD	Artifact uploading user’s password
ARTIFACTORY_REPO	Repo name.(ex: myrepo)
ARTIFACTORY_USER	Artifact uploading user’s username

Step 5.2 - Setup Artifact build and upload job¶

Create a jenkins Pipeline project.
Use this script for the pipeline.

Note

Jenkins server default workspace URL is /var/lib/jenkins/workspace/. If you have installed the Jenkins server in a different location, change this path in the script accordingly.
Under the Build Triggers set the GitHub hook trigger for GITScm polling.
Save the configuration. Now the Jenkins job is configured to listen to any change in the source repository and upload any new update to the artifact repository.

To set up, Execute a build job. Note that this will fail.

Step 5.3 - Setup Artifact deployment Job¶

The artifact deployment job will listen to any new updates in the Artifactory repository and deploy the artifact with the configurations in the Deployment git repository to the dev API-M environment.

Create a Jenkins pipeline using this script .
Save the script and execute a build to setup. Note that this will fail.
To test whether the webhook between Artifactory and the Jenkins job is working, you could execute the same upload test we did under Setup JFrog Artifactory section
```
$ touch resource.txt
$ curl -u repouser:<password> -X PUT https://xxxxx.jfrog.io/artifactory/myrepo/ -T resource.txt
```

If the configuration is working, this will trigger a new build.

Test the scenario¶

To test this setup, use the OpenAPI definition based method to create the APIs.

Create API using an OAS Definition. This will create a project with name PetstoreAPI. A sample command is given below.

$apictl init PetstoreAPI --oas https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v2.0/yaml/petstore.yaml

Generate deployment resource using the command given below.

$apictl gen deployment-dir --source /Path/To/PetstoreAPI --destination <deployment_repo_path>/poc-cicd-deployment-repo

You will see the resources created with the following structure.

├── DeploymentArtifacts_SwaggerPetstore-1.0.0
│   ├── api_meta.yaml
│   ├── certificates
│   └── params.yaml

Open the params.yaml file and update the content with the code given below.

environments:
  - name: dev
    configs:
      endpoints:
          production:
              url: 'https://petstore.swagger.io/v2/'
      deploymentEnvironments:
          - displayOnDevportal: true
            deploymentEnvironment: Default
            deploymentVhost : localhost

The environment name is set as dev. This configuration will be used for the dev environment.

Commit this to the dev branch in the deployment repository.
Copy the PetstoreAPI folder to the source repository git location.
Create a file named meta.yaml inside the PetstoreAPI folder and add a version similar to following.
```
version: 1.0.0
```
This version is used to deploy the bundle created using the PetstoreAPI API in the Artifactory repository.
Following is the content for the PetstoreAPI. Commit all the files to the git repository
```
├── PetstoreAPI
│   ├── Definitions
│   │   └── swagger.yaml
│   ├── README.md
│   ├── api.yaml
│   ├── api_meta.yaml
│   ├── deployment_environments.yaml
│   └── meta.yaml
```
When you commit to the source repository, the CICD_ARTIFACT_UPLOAD will get triggered first and upload the built component to the Artifactory. This is displayed in the console log of the Jenkins job.

Once the bundle is uploaded to the Artifactory repository, it will trigger the deployment Jenkins job. The bundle getting deployed in API-M is displayed in the logs.

Log in to the API Manager instance and you should see the API is created.

Let's update the API and see the changes. The API is in CREATED state. Let’s publish this API by following the steps given below.

Open the PetstoreAPI/api.yaml file and change the lifeCycleStatus to PUBLISHED
Open PetstoreAPI/meta.yaml file and change the version to some new value (let’s say 1.0.1) .
Commit both files.

Once you commit the changes to the source repository, you would see the jenkins jobs getting triggered. Log in to the Publisher Portal and check the changes.

If you go to the artifactory repository and check the PetstoreAPI you will see two versions in the repository.

Top