

CI/CD with WSO2 API Manager¶

APIs have become a defacto for connecting apps, services, and data. An organization can have multiple environments, such as development, testing, QA, staging, and production, each with its own instance of API Managers. Therefore, the APIs need to be available in each environment after developers specify the required conditions. Manually promoting APIs between environments is a tedious, error-prone, and time-consuming task. This drastically reduces an organization’s productivity.

WSO2 API Manager addresses the issue of API automation by providing a platform-agnostic, developer-centric solution. WSO2 API Controller, apictl tool plays a key role in the automation pipeline. It can seamlessly integrate environment-related configurations and also create API Projects from OpenAPI specifications, opening a gate to fully automated API deployment with only a few steps. With the power of flexible tooling, WSO2 API Manager is ready to address modern requirements for automating API deployments.

Continuous integration and continuous deployment for APIs is an open-ended scenario; different organizations have different ways of addressing the problem. The above diagram depicts a generic solution that involves a minimum number of parties in an organization for API automation. Although the diagram shows three parties, there could be more or less depending on the organization’s structure.

API Developers: Develop APIs and related services
API Publishers: Publish APIs to users
DevOps: Control the deployment process

API Developers and Publishers work with a version control system, which acts as a single source of truth for the pipeline.

API Publisher based CI/CD¶

API Developers can use the API Publisher in WSO2 API Manager to create APIs in a lower environment. CI/CD for API Manager relies on a Version Control system that acts as a Single Source of Truth for the pipeline. Therefore, after the API Developer exports the APIs from the lower environment, the API Developer can commit the exported API artifacts to a source code management repository, run the tests in the lower environment, promote the APIs to an upper environment and generate keys for promoted APIs. This process of promoting the API seamlessly to multiple environments can be automated via the apictl tool and other CI/CD tools (e.g., Jenkins, GitHub). The apictl tool makes this process simpler as it handles per environment-related configurations.

To migrate the existing APIs using the API Publisher via CI/CD carry out the steps mentioned in A, B, C, E, F and G, which is listed under the Building blocks for creating a CI/CD pipeline section, in sequential order.

Dev First based approach based CI/CD¶

WSO2 API Manager supports OpenAPI/Swagger specifications to create APIs. The apictl can generate projects with Swagger/OpenAPI specifications without using the API Publisher in WSO2 API Manager. This powerful feature can be used to design pipelines that depend on Swagger/OpenAPI specifications.

Based on the API Project generation, a powerful pipeline for API automation can be developed using OpenAPI/Swagger. This allows rapid API development and increases developer productivity.

To migrate APIs using the Developer First approach via CI/CD carry out A, D, E, F and G, which is listed under the Building blocks for creating a CI/CD pipeline section, in sequential order.

Building blocks for creating a CI/CD pipeline¶

Let us check out the basic building blocks for creating a CI/CD pipeline with WSO2 API-M in sequential order.

Step A - Prepare the environments¶

Download and install WSO2 API Manager 3.2.0 in your environments.

For more information, see installation Prerequisites.
Download and setup WSO2 API Controller 3.2.0 version, apictl.

For more information, see Download and initialize the ctl tool.

Add API Manager environments using the add-env command.

Example

Linux/Unix

apictl add-env -e dev \
            --apim https://localhost:9443 

apictl add-env -e prod \
            --apim https://localhost:9444

Mac/Windows

apictl add-env -e dev --apim https://localhost:9443 

apictl add-env -e prod --apim https://localhost:9444

For more information, see Add an environment.

Step B - Create and Publish an API in a lower environment¶

Now, you have added two different environments. Our end goal is to automate the API migration between the dev and prod environments. Therefore, first, the API should be published in the dev environment using the API Publisher in WSO2 API Manager. For more information on deploying an API in the API Manager, see the Quick Start Guide.

For this example, let's use the Swagger Petstore - OpenAPI 3.0.

Sign in to the API Publisher.

https://localhost:9443/publisher
Click CREATE API and select I have an existing REST API.
Create the API and define its name as SwaggerPetstore and Version as 1.0.0.
Enter the backend URLs for the Production and Sandbox environments, and publish the API.

For this example, let's use the following endpoints.
```
Production: http://dev.wso2.com
Sandbox: http://dev.sandbox.wso2.com
```

Step C - Export an API from a lower environment¶

The apictl can export an API as an archive from a lower environment (i.e., dev), which contains all the information to recreate the API on another upper environment (i.e., prod).

Sign in to the API Manager in the lower environment via the apictl.
Example
```
apictl login dev -u admin -p admin -k
```
For more information, see Login to an Environment.

Tip

A user with admin role is allowed to export APIs. To create a custom user who can export APIs, refer Steps to Create a Custom User who can Perform API Controller Operations.
Export the API from the lower environment using the export-api command.
Example
```
apictl export-api -e dev -n SwaggerPetstore -v 1.0.0 --provider admin
```
For more information, see Export an API.
Extract the content (API will be exported as an archive to the <USER_HOME>/.wso2apictl/exported/apis/dev/ directory). After extraction, you will find a directory named SwaggerPetstore-1.0.0. Rename it to SwaggerPetstore for easy reference.

Step D - Initialize the project using a Swagger/OpenAPI specification¶

Execute the following command to directly generate the PetstoreAPI project using a Swagger/OpenAPI specification. (You can download the Swagger/OpenAPI specification from here.)

Example

apictl init PetstoreAPI --oas path/to/petstore.yaml

This generates an API project in the PetstoreAPI directory using the provided specification. This project can be directly imported into the API Manager.
The apictl allows further customization to the project initialization using a template file. Organization-specific common details can be added into this template file and shared across developers to increase productivity.
To further finetune API creation, an additional API Definition file can be used. This definition file supports detecting environment variables during the creation process. It can be combined with scripting to develop powerful tools for automating API Project creation.
Using this method, the Swagger/OpenAPI specification becomes a single source of truth for API deployment. By combining templating and the definition file, the automation servers can be configured to initialize API Projects from Swagger/OpenAPI specifications and also have custom parameter files. This reduces human intervention and boosts productivity.
For example, when an organization depends on a microservices architecture, this method can be utilized to create an automated pipeline to move Swagger/OpenAPI specifications to upper environments.

For more information on initializing an API Project using OpenAPI/Swagger Specification, see Initialize an API Project.

Step E - Prepare an API project for CI/CD¶

Copy this directory into your Version Control Repository.
- If you are using the Dev First approach - Copy the initialized project directory.
- If you are using the Publisher based approach - Copy the extracted project directory.
Define the environment-specific details in the api_params.yaml parameter file.

Define the prod.wso2.com and prod.sandbox.wso2.com as the backend URLs in this file.

It is recommended to store the parameter file with the API Project; however, you can store it anywhere as required and provide the location to this file as a flag when using the import-api command.
Example
```
environments: 
  - name: dev 
    endpoints: 
      production: 
        url: 'http://dev.wso2.com' 
      sandbox:
        url: 'http://dev.sandbox.wso2.com' 
  - name: prod
    endpoints:
      production:
        url: 'http://prod.wso2.com'
      sandbox:
        url: 'http://prod.sandbox.wso2.com'
```
Info
- The tool reduces the pipeline’s complexity and provides a simple and powerful mechanism to handle environment-specific configurations.
- You can define both production and sandbox backend endpoints and additional configurations for the environments such as retry/suspend timeouts, gateway environments, etc. in the api_params.yaml file.
- Backend certificates for each URL can be configured. For certificates, a valid path to the certificate file is required. These paths can be stored in the Automation Server.
- The apictl supports detecting environment variables defined in usual notation. If an environment variable is not set, the tool will fail. Also, the system will request the user for a set of required environment variables to ensure that information is not missing during the migration process.
- It is recommended to store API and environment-specific parameters in separate repositories.
- For more information on using an environment parameter file, see Configuring Environment Specific Parameters.
Commit the project to the version control system.

Step F - Import the API to an upper environment¶

The Automation Server can be configured to run a specific pipeline for promoting artifacts to other environments. The DevOps team can develop this pipeline further to include automated tests, workflow approvals, and other tasks.

The apictl tool should be installed in the automation servers to begin the process. As the tool supports a variety of platforms, including Linux/Windows and macOS, this can be done easily.

a. Promoting APIs in a Git repository to upper environments via CI/CD¶

The repository that you committed the project in the above step E needs to be cloned into the instance that is executing the CI/CD process. From 3.2.0 onwards, apictl has the inbuilt support to integrate with a Git based version control system. It gives a unified command vcs deploy to deploy any type of project (e.g., APIs, API Products, and Apps).

Navigate to the directory that has the cloned Git repository.
```
$ cd <cloned-repository-name>
```
As this is the first time that the repository is used for Git Integration functionality of API Controller, run vcs init command to initialize the repository with API Controller. This needs to run only once for the repository.
Example
```
$ apictl vcs init
```
Response: Successfully initialized GIT repository
Once vcs init command is executed, a new file vcs.yaml will be created in the root location of the repository.

Make sure to commit this file to the repository.

Tip

vcs.yaml will contain a unique identifier for the repository which is used to store deployed commits related meta-information.
Run vcs status command to see the available changes that needs to be deployed to the production environment.
Example
```
$ apictl vcs status -e prod
```
Response: Projects to Deploy (1)

APIs (1) ... 1: [save] SwaggerPetstore-1.0.0: (SwaggerPetstore-1.0.0)
Import the SwaggerPetstore API into the production environment by running the following sample command.
Tip
- Make sure you have already logged-in to the prod environment. For more information, see Login to an Environment.
- A user with admin role is allowed to import APIs. To create a custom user who can import APIs, see Steps to Create a Custom User who can Perform API Controller Operations.
Example
```
$ apictl vcs deploy -e prod
```
Response: Deploying Projects (1)...

APIs (1) ... 1: SwaggerPetstore-1.0.0: (SwaggerPetstore-1.0.0) Successfully imported API
The above command will detect the target environment and provision the API to it.

If you run vcs deploy command again, you will see the following output indicating that the deployment is already up-to-date.
Example
```
$ apictl vcs deploy -e prod
```
Response: Everything is up-to-date
Adding a new API to the Git repository

Multiple APIs can be promoted through CI/CD by committing the respective API projects to the repository.
1. Create another API Project (Pizzashack-1.0.0) by following the steps (B,C) OR D.
2. Commit the project to the Git repository.
3. Run vcs status command to verify the new project addition.
```
$ apictl vcs status -e prod
```
  Response: Projects to Deploy (1)
  
  APIs (1) ... 1: [save] Pizzashack-1.0.0: (Pizzashack-1.0.0)
4. Run vcs deploy command to deploy the new API to the production environment.
```
$ apictl vcs deploy -e prod
```
  Response: Deploying Projects (1)...
  
  APIs (1) ... 1: Pizzashack-1.0.0: (Pizzashack-1.0.0) Successfully imported API
  
  Here, apictl will deploy only the new API Pizzashack-1.0.0 without re-deploying the other unchanged API SwaggerPetstore-1.0.0.
Important

For deploying an API using vcs deploy command:
- It is mandatory to have your API projects in a Git based version control system.
- It is mandatory to have api_params.yaml file inside each API Project. This is created by default when you export an API using export-api or initialized an API Project using init. The following configuration section in the api_params.yaml file is used to deploy the API.
```
deploy:
    import:
        update: true
        preserveProvider: true
```
You can change the above fields accordingly.

Field Description

update Used to specify whether to update the API if it already exists during the deployment

preserveProvider Preserve existing provider of API after importing it

b. Promoting a single API via CI/CD to upper environments¶

You can use the following alternative approach to promote a single API via CI/CD. This method is ideal if your CI/CD pipeline is not built based on Git.

Import the SwaggerPetstore API into the production environment and test the API by running the following sample command.
Example
```
apictl import-api -f ./SwaggerPetstore -e prod --preserve-provider=false --update=true
```
Note
- When the update flag is present, WSO2 API Manager will attempt to seamlessly update if an existing API is found with the same name and version.
- The import command prepares an API Project for WSO2 API Manager by processing the parameter file. It determines which configuration should be processed to create an API Project by detecting the environment that has been used to import it.
- For more information on importing an API to an environment, see Import an API.
The above command will detect the target environment and provision the API to it.
Sign in to the API Publisher.

https://localhost:9444/publisher
Check the details of the API.

You will see that the API has been imported with correct environment-specific details that you defined. - If you have followed, A, B, C, E, and F, then you can see that your API is in the PUBLISHED state. - If you have followed A, D, E, and F, then you can see that your API is in the CREATED state.

Info

When exporting an API, the apictl tool will also export the API’s lifecycle status. When importing to another environment, this lifecycle status will be preserved. This ensures that the API has the same state across environments.
For example, if an API is in the PUBLISHED state in the development environment, it will also be in the same state in the testing environment. This default behavior can be changed via the apictl tool, which assigns APIs the CREATED state after importing.

Step G - Get keys for an API/API Product¶

Follow the instructions below to generate a JWT/OAuth token for testing purposes using CTL in order to invoke an API or an API Product by subscribing to it using a new application created by CTL:

Tip

Make sure that WSO2 API Manager is started and the CTL tool is running. For more information, see Download and Initialize the CTL Tool.
You should log in to the API Manager in the environment by following the instructions in Login to an Environment.

Run any of the following CTL commands to get keys for the API/API Product.

Command
```
apictl get-keys -n <API or API Product name> -v <API version> -r <API or API Product provider> -e <environment> -k
```
```
apictl get-keys --name <API or API Product name> --version <API version> --provider <API or API Product provider> --environment <environment> -k
```
Example
```
apictl get-keys -n PizzaShackAPI -e dev -k
```
Example
```
apictl get-keys -n PizzaShackAPI -v 1.0.0 -e dev -k -r admin -t https://localhost:8243/token
```
Info

Flags:
- Required :
  --environment or -e : Key generation environment
  --name or -n : API or API Product to generate keys for
- Optional :
  --token or -t : New token endpoint of the environment (This overrides the previously provided token endpoint that was provided using the add-env command)
  --provider or -r : Provider of the API or API Product
  --version or -v : Version of the API (Currently API Products do not have versions)
Note

Both the flags (--name (-n) and --environment (-e)) are mandatory. You can override the given token endpoint or the default token endpoint using the --token (-t) optional flag together with the new token endpoint.

Info

Upon running the above command, the CTL tool will create a default application in the environment, subscribe to the API, and generate keys based on the token type defined in the <USER_HOME>/.wso2apictl/main-config.yamlfile.
Using apictl tool the HTTP request timeout, and export directory can be set up and changed. For more information on changing the HTTP request timeout, see Set HTTP request timeout and Set export directory accordingly.
When running the above command, if you have not specified the --version (-v), the tool will consider the version as 1.0.0 by default. If you have specified the version, then that value will be considered.

Step H - Extending a CI/CD pipeline to support API Products¶

For example, let's consider there is an API Product PetsInfo in the development environment with a subset of operations of SwaggerPetstore API.

Export the API Product using export api-product command from the development environment (dev).
```
$ apictl export api-product -n PetsInfo -e dev
```
Response: Successfully exported API Product! Find the exported API Product at /home/wso2user/.wso2apictl/exported/api-products/dev/PetsInfo_1.0.0.zip
Extract the exported API Product Project.
Commit the project to the Git repository.
Run vcs status command to see the available changes that needs to be deployed to the production environment.
Example
```
$ apictl vcs status -e prod
```
Response: Projects to Deploy (1)

API Products (1) ... 1: [save] PetsInfo-1.0.0: (PetsInfo-1.0.0)
Important

If you haven't initialized the repository with API Controller, you will get the below error.
```
$ apictl vcs status -e prod
```
Response: apictl: The repository info: vcs.yaml is not found in the repository root. If this is the first time you are using this repo, please initialize it with 'vcs init'. Exit status 1

Make sure to follow Promoting APIs in a Git repository to upper environments via CI/CD - Step 2 to initialize the repository.

Import the PetsInfo Product into the production environment by running the following sample command.

Example

$ apictl vcs deploy -e prod

Response: Deploying Projects (1)...

API Products (1) ... 1: PetsInfo-1.0.0: (PetsInfo-1.0.0) Successfully imported API Product

The above command will detect the target environment and create the PetsInfo Product in the target environment.

Important

For deploying an API Product using vcs deploy command:

It is mandatory to have your API Product projects in a Git based version control system.
It is mandatory to have api_product_params.yaml file inside each API Product Project. This is created by default when you export an API Product using export api-product. The following configuration section in the api_product_params.yaml file is used to deploy the API Product.

deploy:
    import:
        updateApiProduct: true
        preserveProvider: true
        importApis: true
        updateApis: false

You can change the above fields accordingly.

Field	Description
updateApiProduct	Used to specify whether to update the API Product if it already exists during deployment
preserveProvider	Preserve the existing provider of the API Product after importing it
importApis	Import the dependant API(s) along with the API Product if the dependant API(s) are not available in the target environment
updateApis	Update the dependant API(s) in the target environment

Tip

Multiple API product projects can be promoted through CI/CD by committing them to the same repository.

Step I - Extending a CI/CD pipeline to support applications¶

Let's assume that the PetsApp application is in the development environment which is already subscribed to the SwaggerPetstore API.

Export the Application using the export app command from the development environment (dev). Note that --withKeys option is used to export the subscriptions and keys (if any) of the application.
```
$ apictl export-app --name PetsApp --owner david -e dev --withKeys
```
Response: Successfully exported Application! Find the exported Application at /home/wso2user/.wso2apictl/exported/apps/dev/david_PetsApp.zip
Extract the exported Application Project.
Commit the project to the same git repository.
Run the vcs status command to see the available changes that need to be deployed to the production environment.
Example
```
$ apictl vcs status -e prod
```
Response: Projects to Deploy (1)

Applications (1) ... 1: [save] PetsApp: (PetsApp)
Important

If you haven't initialized the repository with API Controller, you will get the below error.
```
$ apictl vcs status -e prod
```
Response: apictl: The repository info: vcs.yaml is not found in the repository root. If this is the first time you are using this repo, please initialize it with 'vcs init'. Exit status 1

Make sure to follow Promoting APIs in a Git repository to upper environments via CI/CD - Step 2 to initialize the repository.

Import the PetsApp Application into the production environment by running the following sample command.

Example

$ apictl vcs deploy -e prod

Response: Deploying Projects (1)...

Applications (1) ... 1: PetsApp: (PetsApp) Successfully imported Application

The above command will detect the target environment and create the application in the target environment.

Important

For deploying an application using vcs deploy command:

It is mandatory to have your Application projects in a Git based version control system.
It is mandatory to have application_params.yaml file inside each application project. This is created by default when you export an Application using export-app. The following configuration section in the application_params.yaml file is used to deploy the application.

deploy:
    import:
        update: true
        preserveOwner: true
        skipSubscriptions: false
        skipKeys: true

You can change the above fields accordingly.

Field	Description
update	Used to specify whether to update the application if it already exists during the deployment
preserveOwner	Preserve existing owner of the application after importing it
skipSubscriptions	Specifies whether to import the subscriptions of the application
skipKeys	Specifies whether to import the credentials of the application

Tip

Multiple applications could be promoted through CI/CD by committing those application projects to the same repository.

Now, you know the building blocks of creating a CI/CD pipeline using apictl. By using the above, you can create an automated pipeline for API promotion between environments using either one of the latter mentioned approaches.

More

Next let's use the above knowledge to create a Jenkins CI/CD Pipeline with WSO2 API Management for a Dev First Approach.

Top