What’s new in WSO2 API Microgateway 3.0 (Part I)

The world famous open-source middleware giant, WSO2s’ API MicroGateway is an enterprise-grade, cloud native and open source API management solution which is fully optimized for microservice platforms. Most importantly, the API MicroGateway is Dev and Dev-Ops friendly, highly scalable, stateless, decentralized, and it is developed on top of Ballerina, the WSO2s’ latest cloud native programming language and also Ballerina is promised lightweight containers along with fast boot-up times, low memory footprint, and small distribution sizes.

Most recently, WSO2 has released the latest version of their stunning MicroGateway which is called “WSO2 API MicroGateway 3.0”. The latest release of API MicroGateway accompanied a set of brand new stunning features including API definition per resource endpoints, HTTP2 support, Mutual SSL based authentication, Config based basic authentication, Global throttling, Request/Response transformation and schema validation, Integration with 3rd party key managers and much more. The latest MicroGateway has updated its architecture to be more developer friendly and also decomposed the runtime and developer tools (MicroGateway toolkit). The MicroGateway 3.0 standardize the API definition by open API specifications which makes easier the developer life by collaborate in creating APIs and testing them independently and it would definitely be a developer-first approach as they introduced in the release notes.

The main objective of this article series is to guide you on installation (with dependencies), make you familiar with the architectural changes and also demonstrate the latest features introduced by the WSO2 API MicroGateway 3.0. It will include practical implementation of different use-cases to demonstrate the latest MicroGateways’ capability by using a customer API metaphor of an online store.

01. Installation Guide

1.1. Java (JDK 1.8)

Java is a general purpose programming language and it is most commonly used in backend servers. Java is used as the underlying runtime engine for most of the WSO2 software as well as WSO2 ballerina and API MicroGateway. The JDK 1.8 is the mostly recommended version for Ballerina and MicroGateway. Please download and install your favorite JDK 1.8 compatible java distribution.

1.2. Ballerina

Please download and install the latest Ballerina version from the WSO2 Ballerina official page. You may also install the IDE plugin into your favorite IDE.

Please refer to the Ballerina installation guide for more information.

1.3. WSO2 API Microgateway  Toolkit and Runtime

As mentioned above, the WSO2 MicroGateway 3.0 has decomposed the distribution of Toolkit and Runtime. You may download the releases from WSO2 official release page  and extract the binaries. Make sure the /bin folder of both MicroGateway Toolkit and Runtime installations are in your PATH variable. Then you can execute micro-gw and “gateway” commands directly from the terminal. (Hint: Please make sure they have got the executable permission (+x) in unix/linux operating systems)

A sample .bash_profile file for unix-like operating systems (you may manually add the installation folders in to the PATH variable of windows operating systems)

# microgateway toolkit directory
export MGW_TOOLKIT_HOME = /opt/wso2-microgateway-toolkit

# microgateway runtime directory
export MGW_RUNTIME_HOME = /opt/wso2-microgateway-runtime

# adding microgateway components into your PATH
export PATH=$PATH:$MGW_TOOLKIT_HOME/bin:$MGW_RUNTIME_HOME/bin

Please refer to the MicroGateway 3.0 official installation guide for more information.

Note:
The WSO2 API MicroGateway Runtime could be deployed in many different patterns. But MicroGateway deployment patterns are not the interest of this stage of the article series. Please refer to the “WSO2 API MicroGateway deployment patterns” article by Chanaka Fernando for further information of complex deployments. However it’s recommended to keep the deployment simple for the purpose of this article

02. Create, build and run a MicroGateway project

2.1. Sandbox Project

The sandbox project contains the required dependencies and code snippets for each scenario of the article. You may start to work on the “sandbox” directory and the final version of each scenario is in the “complete” directory. Please clone the sandbox project from the GitHub repository.

Note: 
You may use your favorite git client or execute following commands to clone the git repository. Please don’t forget to start the sandbox project.

git clone git@github.com:hjchanna/wso2-api-micro-gw3-sandbox.git

Sandbox project also contains the ballerina implementation of the customer service for the online store metaphor as mentioned in above sections. The customer service contains following resource endpoints which would use to demonstrate MicroGateway capabilities in later sections.

Please follow the steps below to start the customer API service.

• Locate into the customer_service folder of the sandbox project
• Execute the following command to run ballerina customer service

ballerina run customer_service.bal

Now the customer service should have started in the port 8080 and you may test the customer API by curl command or you may use your favorite API testing tool ie: Postman.

# get all customer data
curl http://localhost:8080/customer

# get customer data by customer id
curl http://localhost:8080/customer/1

# add new customer
curl -X POST http://localhost:8080/customer -d '{"id": "6", "name": "Harry James Potter"}'

2.2. Creating a MicroGateway project

The WSO2 MicroGateway toolkit enables to initialize a project easily with all required configurations. Please execute the following command to generate a new MicroGateway project.

micro-gw init <your-project-name>

Now you should have generated a new MicroGateway project with following structure

<your-project-name>/
├── api_definitions             # open API definition files    
├── conf                        # project config
│   └── deployment-config.toml
├── extensions                 
│   ├── extension_filter.bal
│   ├── startup_extension.bal
│   └── token_revocation_extension.bal
├── interceptors                # custom interceptors
├── policies.yaml
├── services                    # gateway specific services
│   ├── authorize_endpoint.bal
│   ├── revoke_endpoint.bal
│   ├── token_endpoint.bal
│   └── user_info_endpoint.bal
└── target                      # build output
    └── gen

Please refer to the WSO2 official documentation for more information on MicroGateway project structure.

Note:
WSO2 MicroGateway doesn’t successfully build a project which doesn’t contain valid open API definitions. Please copy the customer_api.yaml file into the api_definitions folder of your project. The customer_api.yaml file is provided by the sandbox projects’ sandbox directory.

2.3. Building and Running a MicroGateway Project

The WSO2 MicroGateway toolkit also enables to build a project easily through the terminal. Please locate into the parent directory of your project and execute the following command to build your microgateway project.

micro-gw build <your-project-name>

The micro-gw build command translates the open API definition files into ballerina source files and compiles those ballerina source files into ballerina executables. The micro-gateway build command also generates the Dockerfile and images if you have configured the project. The build output file should be available in following location after a successful build.

<your-project-folder>/target/<project-name>.balx

Once you build the MicroGateway project, you may run the project through the microgateway runtime command as follows.

gateway <your-project-folder>/target/<project-name>.balx

Now your MicroGateway should have started in port numbers 9090, 9095 and 9096 by default. MicroGateway default ports are utilized as follows.

Port 9090 - exposes API through HTTP protocol
Port 9095 - exposes API through HTTPS protocol
Port 9096 - MicroGateway management API through HTTPS protocol

Note:
Please note that following project conventions will be used in the later sections of the article.

$MGW_TOOLKIT_HOME - microgateway toolkit directory
$MGW_RUNTIME_HOME - microgateway runtime directory
$PROJECT_HOME - <sandbox project dir>/sandbox directory

03. Config based “Basic authentication”

The basic authentication scheme a standard simple authorization scheme uses base64 encoded authorization header with username and password. Out of the box, WSO2 MicroGateway enables APIs to authorize through basic authentication scheme in both API level as well as the resource level. Following step by step guidance will walk you through both API level and resource level configuration. You may tryout basic authentication in either API level (section A) or resource level (section B) or both separately.

3.1. Define basic authentication in open API (API level)

• Follow the steps above to initialize a new MicroGateway project with name basic_auth_sample and copy the sample open API definition file (customer_api.yaml file provided by the sandbox project) into the api_definition folder of your project.

• Add following basic authentication configuration into the newly created online_store.yaml

# enable the basic authentication scheme for the API
security:
  - online_store_basic_auth: []
# define the basic authentication scheme
components:
  securitySchemes:
    online_store_basic_auth:
      type: http
      scheme: basic

3.2. Define basic authentication in open API (Resource level)

• Initialize new MicroGateway project with name basic_auth_sample and copy customer_api.yaml open API definition file into the api_definition folder of your project.

• Add basic authentication configuration into the each resource of newly created customer_api.yaml file

Add security section into each resource as follows

...
  "/customer":
    get:
      responses:
        "200":
          description: ok, get all customer data
      security:
        - online_store_basic_auth: []
...

Define the basic authentication scheme under the component section components:

 securitySchemes:
    online_store_basic_auth:
      type: http
      scheme: basic

3.3. Configure MicroGateway for Basic Authentication

• Open the MicroGateway configuration file ($MGW_RUNTIME_HOME/conf/micro-gw.conf) in your favorite editor
• Locate into [b7a.users] section and add following basic authentication configuration

["b7a.users"]
["b7a.users.<username>"]
password="<SHA1 hash password>"

Ex: Username: admin, Password: password
["b7a.users"]
["b7a.users.admin"]
password="5BAA61E4C9B93F3F0682250B6CF8331B7EE68FD8"

Note:
The password should be SHA1 encrypted value in uppercase. You may use an online tool to generate the hash or generate through the terminal by the following command.

echo -n "<password>" | openssl sha1 | tr 'a-z' 'A-Z'

3.4. Build, run and test the project

• Use MicroGateway toolkit to build the basic_auth_sample project as mentioned in “Before you begin” section (micro-gw build basic_auth_sample). Once the build is successful it should have generated the customer_api.balx file in the target directory of the project.

• Start the MicroGateway with newly generated customer_api.balx file.
  gateway $PROJECT_HOME/basic_auth_sample/target/customer_api.balx
  The WSO2 MicroGateway will be started in the ports 9090, 9095, and 9096 and it is time to test the API.

• The micro-gw.conf file was configured the basic auth credentials to be “admin” and “password”. According to the basic authentication scheme, the “Authorization” header needs to be base64 encrypted username and the password separated by a colon. Generate base64 for the string “admin:password” through the terminal.
  echo “admin:password” | base64

• Use curl command or your favorite API testing tool to test the API

curl -X GET \
 http://localhost:9090/api/v1/customer \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ='

And also the HTTPS endpoint also available to access and use the curl command as follows to test the HTTPS endpoints. (Note: HTTPS security will be ignored in this stage since the certificates are not properly configured).

curl -kX GET \
 http://localhost:9095/api/v1/customer \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ='

04. Request and Response schema validation

The WSO2 MicroGateway enables to configure request and response schema validation through the open API definition. It is required to simply define the request or response schema in the open API and enable the schema validation through the MicroGateway runtime configuration.

4.1. Open API Configuration for request/response schema validation

Follow the steps below to tryout schema validation of a WSO2 MicroGateway project.

• Initialize a new WSO2 MicroGateway project through MicroGateway toolkit with the name “request_response_validation”. And copy the customer_api.yaml into api_definitions folder of the project.

• Configure the request_response_validation API to authorize through basic authentication scheme as mentioned in the above section.

• Add customer schema into the component section of the open API definition file (customer_api.yaml)

components:
     ...
  schemas:
    customerSchema:
      type: object
      required:
        - id
        - name
      properties:
        id:
          type: integer
        name:
          type: string

• Define the open API request body for POST /customer endpoint in the component section of the open API definition file and link to the relevant endpoint

components:
     ...
  requestBodies:
    customerBody:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/customerSchema'
      required: true

Once you defined the request body for POST /customer endpoint it is need to be linked into the relevant endpoint definition. Add following lines into the post section of  the /customer endpoint as follows.

post:
...
  requestBody:
        $ref: '#/components/requestBodies/customerBody'

It should have successfully configured schema validation for the request body of POST /customer endpoint. You may jump into the test section or continue into the next section of schema validation configuration of the GET /customer/{customerId} and GET /customer endpoints.

• Since the customer schema is already defined in the above step, it is possible to link the customer schema directly into the GET /customer/{customerId} response. Add following config into the response 200 section of the relevant endpoint.

'200':
     ...
         content:
           application/json:
              schema:
                $ref: '#/components/schemas/customerSchema'

• A successful response of GET /customer endpoint returns a json array of customer objects. Therefore it is required to define the response schema to be an array of customer objects. Add following configurations into the GET /customer endpoint in order to define the response schema.

get:
...
       responses:
        '200':
          ...
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/customerSchema'

• It is also possible to validate path variables and query parameters of a request using the MicroGateway. In this example we will consider the request validation of GET /customer/{customerId} endpoint. The expect the customerId to be an integer path variable and it could be defined as follows.

get:
...
  parameters:
        - in: path
          name: customerId
          required: true
          schema:
            type: integer
            format: int32

4.2. MicroGateway runtime configuration

You may need to enable request and response schema validation in the MicroGateway runtime. Please open the MicroGateway config file and locate into “[validationConfig]” section. Update enableRequestValidation and enableResponseValidation configurations as follows.

[validationConfig]
enableRequestValidation = true
enableResponseValidation = true

4.3. Build, run and test the project

• Build the project using MicroGateway toolkit as follows (Please refer to the build, run and test section of the above example for more detailed information)

micro-gw build request_response_validation

• Start the MicroGateway with the newly generated ballerina executable file

gateway \
$PROJECT_HOME/target/request_response_validation.balx

MicroGateway should have started in default ports and continue to the next step to test each scenario of the use-case.

• Test each scenario of schema validation by modifying the values of request params and the request body.

request parameter validation of GET /customer/{customerId} endpoint

curl -X GET \
 http://localhost:9090/api/v1/customer/{integer-customer-id} \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ='

request body validation for the customer schema

curl -X POST \
 http://localhost:9090/api/v1/customer \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
 -d '{"id":1,"name": "Harry James Potter"}'

05. Request and Response transformation

WSO2 API MicroGateways enables the interceptor pattern which is supportive to extend MicroGateways’ functionality. MicroGateway interceptors will be defined as ballerina function, and it is capable to conduct mediations and transformations. There are two types of interceptors available in the MicroGateway which are request interceptors and respond interceptors. Request interceptors are gets triggered before sending the request to the backend service while response interceptors are gets triggered before sending response to the API client.

As mentioned earlier, interceptors should be defined in ballerina language and respective ballerina file has to be placed in the interceptors directory of a MicroGateway project. And also, interceptor should be linked into the API in open API definition file either in API level or resource level through the WSO2 specific open API extensions as follows.

#request interceptor
x-wso2-request-interceptor: <your-request-interceptor>

#response interceptor
x-wso2-response-interceptor: <your-response-interceptor>

5.1. Request Interceptor sample

According the sandbox customer API, POST /customer endpoint is used to add a new customer. However the request body has to have the customer id defined, which is unpredictable from the clients’ point of view and also the customer id is hard to maintain in a sequence order. In this use-case will walk you through the automated customer id generation and transform the request payload into accepted format by the backend service.

• Initialize a new MicroGateway project with and copy provided open API definitions into the project. The project would be “request_response_transformation”.
• Configure the newly created MicroGateway to enable basic authentication scheme.
• Create a new ballerina file with name “transform_post_request.bal” in the interceptors directory of the MicroGateway project and copy following ballerina code into the transform_post_request.bal

import ballerina/log;
import ballerina/http;

//customer API client
http:Client httpClient = new("http://localhost:8080");

public function transformPostRequest (http:Caller outboundEp, http:Request req) {

    //get all customers to calculate next customer id
    var response = httpClient->get("/customer");
    var id = -1;

    if (response is http:Response) {
        json | error customers = response.getJsonPayload();

        if (customers is json) {
            //calculate next customer id
            id = customers.length() + 1;
        }
    }

    log:printInfo("Generated a new id for the customer :" + id);

    //set the calculated customer id to the request payload
    var requestPayload = req.getJsonPayload();
    if (requestPayload is json) {
        requestPayload.id = id;
        req.setJsonPayload(untaint requestPayload, contentType = "application/json");
    }
}

Above ballerina code will talk to the GET /customer endpoint of the backend customer service and gets all customers. And it will generate the next customer id by adding one integer into the number of customers returned. After all, it will add or update the customer id in the request body and will pass into the backend.

Note:
It is not really recommended to call the backend service bypassing the API gateway and it is implemented just for demonstration purposes only.

• Link the request interceptor to the relevant resource of the API definition file using WSO2 specific open API extension as follows and it is ready to build and test.

post:
  ...
  x-wso2-request-interceptor: transformPostRequest

5.2. Response Interceptor sample

You might have noticed the customer backend service returns an error message in json format when it is in error scenarios. For an example GET /customer/{customerId} endpoint will return following response if there is not a customer available for the customer id provided.

{
     “message” : ”Customer not found for id: <customer-id>”
}

This section will walk you through the transforming such responses into the standard error message format of the API MicroGateway with the following format.

{
    "fault": {
        "code": 900000,
        "message": "Customer not found for the id: <customer-id>",
        "description": "An error returned from the customer backend"
    }
}

Let’s continue to implement a new ballerina response interceptor on top the of request_response_transformation project.

• Create a new ballerina file with the name “transform_message_response.bal under the interceptors folder of the project. And add following ballerina code into the ballerina file.

import ballerina/http;

public function transformMessageResponse (http:Caller inboundEp, http:Response res) {
    //get and check the response code from the backend
    int statusCode = res.statusCode;

    if (404 == statusCode || 500 == statusCode) {
        //create a new json response payload to return to the client
        json responsePayload = {

        };

        json | error backendPayload = res.getJsonPayload();
        //transform the payload into the required format

        if (backendPayload is json) {
            responsePayload.fault = {};
            responsePayload.fault.code = 900000;
            responsePayload.fault.message = backendPayload.message;
            responsePayload.fault.description = "An error returned from the customer backend";

            //set the response payload
            res.setJsonPayload(untaint responsePayload, contentType = "application/json");
        }
    }
}

Above ballerina code will check the response code returned from the backend service and will transform the response payload into required format if the response code is equal to 404 or 500.

• Link the response interceptor in the open API definition in API level which will be applied to all responses as follows.

#add following line into the root level of the open API def
x-wso2-response-interceptor: transformMessageResponse

5.3. Build, run and test the project

• Build the project using MicroGateway toolkit as follows (Please refer to the build, run and test section of basic authentication example for more detailed information)

micro-gw build request_response_transformation

• Start the MicroGateway with the newly generated ballerina executable file

gateway \
$PROJECT_HOME/target/request_response_tranformation.balx

MicroGateway should have started in default ports and continue to the next step to test each scenario of the use-case.

• Test the add new customer endpoint without customer id in the request body. The customer id suppose to automatically generate by the MicroGateway interceptor itself.

curl -X POST \
 http://localhost:9090/api/v1/customer \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ=' \
 -d '{"name": "Harry James Potter"}'

Once above request successfully completed, call GET /customer endpoint to verify the result.

curl -X GET \
 http://localhost:9090/api/v1/customer \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ='

Verify that the newly added customer should have added  to the backend with the next available customer id.

• Test GET /customer/{customerId} endpoint for a not existing customer id and it should return an error message with the expected error format.

curl -X GET \
 http://localhost:9090/api/v1/customer/100 \
 -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQ='

Summary

I hope this article will help you to get stand with the WSO2 API MicroGateway 3.0 including installation, latest architectural style, and also to try out several new features introduced by the latest version of API MicroGateway. As I have demonstrated some of the newly introduced MicroGateway features in a very simple deployment of the MicroGateway runtime, there are much more to present you including Key manager configuration, Throttling, SSL Configuration, HTTP/2 support, Docker/Kubernetes deployments, Runtime service discovery, and Composing multiple microservices which requires a complex MicroGateway deployment to demonstrate. I will leave such use-cases into the next episodes of the article series to prevent the article getting more lengthy and complicated.

I think you enjoyed the first episode of the article series and don’t forget to leave your kind response. Please contact me if you have any queries which regards to the WSO2 API MicroGateway, I am happy to hear from you.