Microsoft Azure Storage Connector Example

Given below is a sample scenario that demonstrates how to work with container and blob operations using the ESB Microsoft Azure Storage Connector.

What you'll build

This example demonstrates how to use Microsoft Azure Storage connector to:

  1. Create a container (a location for storing employee details) in Microsoft Azure Storage account.
  2. Retrieve information about the created containers.
  3. Upload text or binary employee details (blob) in to the container.
  4. Retrieve information about the uploaded employee details(blob).
  5. Remove uploaded employee details (blob).
  6. Remove created container.
  7. Retrieve the metadata from a specific file (blob).

All seven operations are exposed via an API. The API with the context /resources has seven resources

  • /createcontainer : Creates a new container in the Microsoft Azure Storage account with the specified container name for store employee details.
  • /listcontainer : Retrieve information about the created containers from the Microsoft Azure Storage account.
  • /adddetails: Upload text or binary employee data (blob) and stored into the specified container.
  • /listdetails : Retrieve information about the added employee data (blobs).
  • /deletedetails : Remove added employee data from the specified text or binary employee data (blob).
  • /deletecontainer : Remove created container in the Microsoft Azure Storage account.
  • /listmetadata : Retrieve the metadata from a file (blob) stored in the Microsoft Azure Storage container.

For more information about these operations, please refer to the Microsoft Azure Storage connector reference guide.

Note: Before invoking the API, you need to create a Storage Account in Microsoft Azure Storage account. See Azure Storage Configuration documentation for more information.

The following diagram shows the overall solution. The user creates a container, stores some text or binary employee data (blob) into the container or the blob metadata, and then receives it back. To invoke each operation, the user uses the same API.

Microsoft Azure Storage Connector

If you do not want to configure this yourself, you can simply get the project and run it.

Configure the connector in ESB Integration Studio

Follow these steps to set up the ESB Solution Project and the Connector Exporter Project.

  1. Open ESB Integration Studio and create an Integration Project. Creating a new Integration Project

  2. Right-click the project that you created and click on Add or Remove Connector -> Add Connector. You will get directed to the Connector Store.

  3. Search for the specific connector required for your integration scenario and download it to the workspace. Search Connector in the Connector Store

  4. Click Finish, and your Integration Project is ready. The downloaded connector is displayed on the side palette with its operations.

  5. You can drag and drop the operations to the design canvas and build your integration logic. Drag connector operations

  6. Right click on the created Integration Project and select New -> Rest API to create the REST API.

  7. Right click on the created Integration Project and select, -> New -> Rest API to create the REST API.

  8. Specify the API name as MSAzureStorage and API context as /resources. You can go to the XML configuration of the API (source view) and copy the following configuration.

    <?xml version="1.0" encoding="UTF-8"?>
<api context="/resources" name="MSAzureStorage" xmlns="http://ws.apache.org/ns/synapse">
    <resource methods="POST" url-mapping="/createcontainer">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <property expression="json-eval($.containerName)" name="containerName" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.createContainer>
                <containerName>{$ctx:containerName}</containerName>
            </msazurestorage.createContainer>
            <log level="full">
                <property name="Container created" value="Container created"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
    <resource methods="POST" url-mapping="/listcontainer">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.listContainers/>
            <log level="full">
                <property name="List containers" value="List containers"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
    <resource methods="POST" url-mapping="/adddetails">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <property expression="json-eval($.containerName)" name="containerName" scope="default" type="STRING"/>
            <property expression="json-eval($.fileName)" name="fileName" scope="default" type="STRING"/>
            <property expression="json-eval($.filePath)" name="filePath" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.uploadBlob>
                <containerName>{$ctx:containerName}</containerName>
                <filePath>{$ctx:filePath}</filePath>
                <fileName>{$ctx:fileName}</fileName>
            </msazurestorage.uploadBlob>
            <log level="full">
                <property name="Uploaded emplyee details" value="Uploaded emplyee details"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
    <resource methods="POST" url-mapping="/listdetails">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <property expression="json-eval($.containerName)" name="containerName" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.listBlobs>
                <containerName>{$ctx:containerName}</containerName>
            </msazurestorage.listBlobs>
            <log level="full">
                <property name="List uploaded emplyee details" value="List uploaded emplyee details"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
    <resource methods="POST" url-mapping="/deletedetails">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <property expression="json-eval($.containerName)" name="containerName" scope="default" type="STRING"/>
            <property expression="json-eval($.fileName)" name="fileName" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.deleteBlob>
                <containerName>{$ctx:containerName}</containerName>
                <fileName>{$ctx:fileName}</fileName>
            </msazurestorage.deleteBlob>
            <log level="full">
                <property name="Delete selected employee details" value="Delete selected employee details"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
    <resource methods="POST" url-mapping="/deletecontainer">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <property expression="json-eval($.containerName)" name="containerName" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.deleteContainer>
                <containerName>{$ctx:containerName}</containerName>
            </msazurestorage.deleteContainer>
            <log level="full">
                <property name="Delete selected container" value="Delete selected container"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
    <resource methods="POST" url-mapping="/listmetadata">
        <inSequence>
            <property expression="json-eval($.accountName)" name="accountName" scope="default" type="STRING"/>
            <property expression="json-eval($.accountKey)" name="accountKey" scope="default" type="STRING"/>
            <property expression="json-eval($.containerName)" name="containerName" scope="default" type="STRING"/>
            <property expression="json-eval($.fileName)" name="fileName" scope="default" type="STRING"/>
            <msazurestorage.init>
                <accountName>eiconnectortest</accountName>
                <accountKey>bWt69gFpheoD6lwVsMgeV5io2/KxlXK1KUcod68PhzuV1xHxje0LBD4Bd+y+ESAOlH5BTAfvdDG5q4Hhg==</accountKey>
            </msazurestorage.init>
            <msazurestorage.listMetadata>
                <containerName>{$ctx:containerName}</containerName>
                <fileName>{$ctx:fileName}</fileName>
            </msazurestorage.listMetadata>
            <log level="full">
                <property name="list  Metadata" value="list Metadata"/>
            </log>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>
</api>
    Note: Please modify the following properties of the configuration as applicable.

  9. As accountKey use the access key obtained from setting up the Microsoft Azure Storage account.

  10. As accountName get the name of created Storage Account inside the Microsoft Azure Storage account.

Now we can export the imported connector and the API into a single CAR application. CAR application is the one we are going to deploy to server runtime.

Exporting Integration Logic as a CApp

CApp (Carbon Application) is the deployable artifact on the integration runtime. Let us see how we can export integration logic we developed into a CApp along with the connector.

Creating Connector Exporter Project

To bundle a Connector into a CApp, a Connector Exporter Project is required.

  1. Navigate to File -> New -> Other -> WSO2 -> Extensions -> Project Types -> Connector Exporter Project.

    Add Connector Exporter Project

  2. Enter a name for the Connector Exporter Project.

  3. In the next screen select, Specify the parent from workspace and select the specific Integration Project you created from the dropdown.
    Naming Connector Exporter Project

  4. Now you need to add the Connector to Connector Exporter Project that you just created. Right-click the Connector Exporter Project and select, New -> Add Remove Connectors -> Add Connector -> Add from Workspace -> Connector

  5. Once you are directed to the workspace, it displays all the connectors that exist in the workspace. You can select the relevant connector and click Ok.

    Selecting Connector from Workspace

Creating a Composite Application Project

To export the Integration Project as a CApp, a Composite Application Project needs to be created. Usually, when an Integration project is created, this project can be created as part of that project by Integration Studio. If not, you can specifically create it by navigating to File -> New -> Other -> WSO2 -> Distribution -> Composite Application Project.

Exporting the Composite Application Project

  1. Right-click the Composite Application Project and click Export Composite Application Project.

    Export as a Carbon Application

  2. Select an Export Destination where you want to save the .car file.

  3. In the next Create a deployable CAR file screen, select both the created Integration Project and the Connector Exporter Project to save and click Finish. The CApp is created at the specified location provided at the previous step.

    Create a deployable CAR file

Get the project

You can download the ZIP file and extract the contents to get the project code.

Download ZIP

Tip

You may need to update the value of the credentials and make other such changes before deploying and running this project.

Deployment

Follow these steps to deploy the exported CApp in the integration runtime.

Deploying on Micro Integrator

You can copy the composite application to the <PRODUCT-HOME>/repository/deployment/server/carbonapps folder and start the server. Micro Integrator will be started and the composite application will be deployed.

You can further refer the application deployed through the CLI tool. See the instructions on managing integrations from the CLI.

Click here for instructions on deploying on ESB Enterprise Integrator 6

  1. You can copy the composite application to the <PRODUCT-HOME>/repository/deployment/server/carbonapps folder and start the server.

  2. ESB EI server starts and you can login to the Management Console https://localhost:9443/carbon/ URL. Provide login credentials. The default credentials will be admin/admin.

  3. You can see that the API is deployed under the API section.

Testing

Invoke the API as shown below using the curl command. Curl Application can be downloaded from here.

  1. Creating a new container in Microsoft Azure Storage for store employee details.

Sample request

`curl -v POST -d {"containerName":"employeedetails"} "http://localhost:8290/resources/createcontainer" -H "Content-Type:application/json"`

Expected Response

 `{
    "success": true
  }`
  1. Retrieve information about the created containers.

Sample request

`curl -v POST -d {} "http://localhost:8290/resources/listcontainer" -H "Content-Type:application/json"`

Expected Response

 It will retrieve all the existing container names.

 `{
     "result":{
        "container":[
           "employeedetails",
           "employeefinancedetails"
        ]
     }
  }`
  1. Upload text or binary employee data (blob).

Sample request

 `curl -v POST -d {"containerName": "employeedetails","fileName": "sample.txt","filePath": "/home/kasun/Documents/MSAZURESTORAGE/sample.txt"} "http://localhost:8290/resources/adddetails" -H "Content-Type:application/json"`

 **Please note :** /home/kasun/Documents/MSAZURESTORAGE/sample.txt should be a valid path to a text file containing employee information.

Expected Response

 `{
      "success": true
  }`
  1. Retrieve information about the uploaded text or binary employee data (blob).

Sample request

 `curl -v POST -d {"containerName": "employeedetails"} "http://localhost:8290/resources/listdetails" -H "Content-Type:application/json"`

Expected Response

 It will retrieve the uploaded text or binary name and the file path.

 `{
      "result": {
          "blob": "http://eiconnectortest.blob.core.windows.net/employeedetails/sample.txt"
      }
  }`
  1. Remove uploaded employee details (blob).

Sample request

 `curl -v POST -d {"containerName": "employeedetails","fileName": "sample.txt"} "http://localhost:8290/resources/deletedetails" -H "Content-Type:application/json"`

Expected Response

 `{
      "success": true
  }`
  1. Remove created container.

Sample request

 `curl -v POST -d {"containerName": "employeedetails"} "http://localhost:8290/resources/deletecontainer" -H "Content-Type:application/json"`

Expected Response

 `{
      "success": true
  }`
  1. Retrieve blob metadata.

Sample request

curl --location --request POST 'http://localhost:8290/resources/listmetadata' \ --header 'Content-Type: application/json' \ --data-raw '{ "containerName": "employeedetails", "fileName":"sample.pdf" }'

Expected Response

{ "result": { "metadata": { "metadataParameter1": "value1", "metadataParameter2": "value2" } } }

Top