Building a CI/CD Pipeline for APIs Using ESB API Controller

In the modern digital-era, many organizations adopt digital transformation technologies to keep up with the changes. APIs are one of the basic building blocks used in a digitally-driven organization. When the number of APIs managed by the organization grows, they need to have an automated process to handle the rapid API development process. Having a proper continuous integration and continuous deployment (CI/CD) process would give an added advantage to your organization.

In this section, you will learn how to build an automated process using the ESB API Controller (apictl) with MWARE ESB. You will see how a solution is built using a version control system (Github), artifact repository(JFrog Artifactory) and CI/CD Tool (Jenkins).

Setting up the environment

Step 1 - Setup Jenkins

Before you begin, you need to set up the Jenkins server on a remote machine which has public access. To use webhooks in Github and Artifactory repository to trigger Jenkins jobs you need a publicly accessible Jenkins server url. If you are trying this on a local machine, you may have to use a tool such as socketxp to create a tunnel.

In this section, the instructions are tested out on an ubuntu server setup on https://cloud.google.com/compute .

Log in to the server and install following.

  1. Install jq using the command given below.
    sudo apt-get install jq
    
  2. Install the apictl tool. Extract to a location henceforth referred as CTL_HOME.

  3. Install java.

  4. Setup the Jenkins server following the instructions given here. You can choose to install the suggested default plugins. This will install the commonly-used plugins.

  5. Register a webhook in the JForg Artifact repository to notify changes. To capture these events and trigger a Jenkins job, install a generic-webhook-trigger plugin.

  6. Configure apictl home as an environment variable. Go to the Manage Jenkins section and select Configure System.

    Under the Global properties select Environment variables and set the APICTL home as below and click Save.

Now the Jenkins server is set up.

Step 2 - Create Github repositories

In this section, two repositories are created to illustrate this scenario.

  1. Source Repository -https://github.com/chamilaadhi/poc-cicd-source-repo

  2. Deployment Repository - https://github.com/chamilaadhi/poc-cicd-deployment-repo

Source repository contains the files and metadata related to the API. Deployment repository contains the configurations related to each environment this APIs is getting deployed.

Step 2.1 - Setup source repository

This section shows how to trigger a Jenkins job when a change is made to the repository. Lets use Github webhooks to trigger a Jenkins job.

  1. Go to the source repository and under the Settings, select Webhooks and add a webhook to the jenkins server.

  1. Use your jenkins server URL (e.g., http://<public_ip>:8080/github-webhook/) as the URL and Content type as application/json.

  2. Under the Which events would you like to trigger this webhook? section, select Let me select individual events, and select pull request and pushes.

Jenkins server URL

Source repository is now configured.

Now let’s initialize the source repository. This will be done from the developer’s local machine.

Note

If you haven’t setup API Controller, set it up using the apictl tool

  1. Clone the source repository. Navigate to the repository folder.

    $ git clone https://github.com/chamilaadhi/poc-cicd-source-repo.git
    $ cd poc-cicd-source-repo
  2. Execute the following command.

    $ apictl vcs init
    This will create a vcs.yaml file in the repository. Commit it to the source repository.

Now the source repository is configured.

Step 2.2 - Setup deployment repository

Deployment repository will be used to gather configurations for each deployment. To separate out the configuration, we will use git branches. Create a branch named ‘dev’ to keep the configuration related to the dev environment.

Step 3 - Setup JFrog Artifactory

Before you begin

Create a free cloud repository using https://jfrog.com/start-free/#saas. After the registration is complete, Login to https://<name>.jfrog.io/ui/

  1. Create a user to upload artifacts. Login using admin credentials and go to Identity and AccessUsers and create a user. All roles are selected in this scenario.

  2. Make the repo accessible public. Go to Security → Settings and select Allow Anonymous Access

  3. Create a repository to upload the artifacts. Go to Repositories and select Add Repositories. Select Generic type and provide a name (e.g., myrepo).

    After saving, You should be able to access https://<your_org>.jfrog.io/artifactory/<repo_name>/.

    Try uploading a sample resource. An example is given below.

    $ touch resource.txt
    $ curl -u repouser:<password> -X PUT https://xxxx.jfrog.io/artifactory/myrepo/ -T resource.txt

    Access the repository URL to see the resource.

  4. Configure a webhook to triggerb the Jenkins job.

    Create a webhook to trigger the Jenkins job when a new resource is uploaded to the repository. Go to GeneralWebhooks and create a new one.

    URL - http://<jenkins_host>:8080/generic-webhook-trigger/invoke?token=123**

    Add the Event set as Artifact was deployed.

    Select the repository from the new window that appears.

Step 4 - Setup API-M instances

Download the MWARE ESB 4.2.0 here and start the instance. Set this in a separate instance as the dev instance.

Step 5 - Configure Jenkins Jobs

Step 5.1 - Setup Global variables

As shown in the Setup Jenkins Section, Go to Manage Jenkins section and select Configure System. Under the Global properties select Environment variables and set following variables with your environment related details.

Variable Description
APIM_DEV_HOST IP or host name of the dev instance (ex 10.100.5.96 )
APIM_PROD_HOST IP or host name of the prod instance (ex 10.100.5.97 )
ARTIFACTORY_HOST Artifactory host name (ex: testrepo.jfrog.io)
ARTIFACTORY_PWD Artifact uploading user’s password
ARTIFACTORY_REPO Repo name.(ex: myrepo)
ARTIFACTORY_USER Artifact uploading user’s username

Step 5.2 - Setup Artifact build and upload job

  1. Create a jenkins Pipeline project.

  2. Use this script for the pipeline.

    Note

    Jenkins server default workspace URL is /var/lib/jenkins/workspace/. If you have installed the Jenkins server in a different location, change this path in the script accordingly.

  3. Under the Build Triggers set the GitHub hook trigger for GITScm polling.

  4. Save the configuration. Now the Jenkins job is configured to listen to any change in the source repository and upload any new update to the artifact repository.

To set up, Execute a build job. Note that this will fail.

Step 5.3 - Setup Artifact deployment Job

The artifact deployment job will listen to any new updates in the Artifactory repository and deploy the artifact with the configurations in the Deployment git repository to the dev API-M environment.

  1. Create a Jenkins pipeline using this script .

  2. Save the script and execute a build to setup. Note that this will fail.

  3. To test whether the webhook between Artifactory and the Jenkins job is working, you could execute the same upload test we did under Setup JFrog Artifactory section

    $ touch resource.txt
    $ curl -u repouser:<password> -X PUT https://xxxxx.jfrog.io/artifactory/myrepo/ -T resource.txt

If the configuration is working, this will trigger a new build.

Test the scenario

To test this setup, use the OpenAPI definition based method to create the APIs.

  1. Create API using an OAS Definition. This will create a project with name PetstoreAPI. A sample command is given below.

    $apictl init PetstoreAPI --oas https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v2.0/yaml/petstore.yaml
    
  2. Generate deployment resource using the command given below.

    $apictl gen deployment-dir --source /Path/To/PetstoreAPI --destination <deployment_repo_path>/poc-cicd-deployment-repo

    You will see the resources created with the following structure.

    ├── DeploymentArtifacts_SwaggerPetstore-1.0.0
    │   ├── api_meta.yaml
    │   ├── certificates
    │   └── params.yaml
    
  3. Open the params.yaml file and update the content with the code given below.

    environments:
      - name: dev
        configs:
          endpoints:
              production:
                  url: 'https://petstore.swagger.io/v2/'
          deploymentEnvironments:
              - displayOnDevportal: true
                deploymentEnvironment: Default
                deploymentVhost : localhost
    The environment name is set as dev. This configuration will be used for the dev environment.

  4. Commit this to the dev branch in the deployment repository.

  5. Copy the PetstoreAPI folder to the source repository git location.
  6. Create a file named meta.yaml inside the PetstoreAPI folder and add a version similar to following.

    version: 1.0.0
    This version is used to deploy the bundle created using the PetstoreAPI API in the Artifactory repository.

  7. Following is the content for the PetstoreAPI. Commit all the files to the git repository

    ├── PetstoreAPI
    │   ├── Definitions
    │   │   └── swagger.yaml
    │   ├── README.md
    │   ├── api.yaml
    │   ├── api_meta.yaml
    │   ├── deployment_environments.yaml
    │   └── meta.yaml
    

    When you commit to the source repository, the CICD_ARTIFACT_UPLOAD will get triggered first and upload the built component to the Artifactory. This is displayed in the console log of the Jenkins job.

    Once the bundle is uploaded to the Artifactory repository, it will trigger the deployment Jenkins job. The bundle getting deployed in API-M is displayed in the logs.

    Log in to the ESB instance and you should see the API is created.

Let's update the API and see the changes. The API is in CREATED state. Let’s publish this API by following the steps given below.

  1. Open the PetstoreAPI/api.yaml file and change the lifeCycleStatus to PUBLISHED
  2. Open PetstoreAPI/meta.yaml file and change the version to some new value (let’s say 1.0.1) .

  3. Commit both files.

Once you commit the changes to the source repository, you would see the jenkins jobs getting triggered. Log in to the Publisher Portal and check the changes.

If you go to the artifactory repository and check the PetstoreAPI you will see two versions in the repository.

Top