

Step 7: Monitor Statistics¶

This step shows how you can monitor the CDC and file statistics of the ESB Streaming Integrator deployment you started and the SweetFactoryApp Siddhi application you created and deployed in the previous steps. For this purpose, you are using the some of the pre-configured dashboards provided by ESB Streaming Integrator. You can host these dashboards in Grafana and view statistics related to ETL activities carried out by the Streaming Integrator. For more information about these dashboards, see Monitoring ETL Statistics with Grafana

Configuring ESB SI to visualize statistics¶

To be able to see visualizations of statistics generated by ESB Streaming Integrator, you are required to download and install Prometheus and Grafana. You need to download the required pre-configured dashboards and import them to Grafana.

Downloading the required dashboards¶

ESB Streaming Integrator provides you with pre-configured dashboards in JSON format. You can import these dashboards to Grafana to view statistics of your Streaming Integrator deployment.

For this scenario, download the following dashboards:

Downloading and setting up Prometheus¶

ESB Streaming Integrator uses Perometheus to expose its statistics to Grafana. Therefore, to download and configure Prometheus, follow the steps below:

Download Prometheus from the Prometheus site. For instructions, see the Prometheus Getting Started Guide.
Extract the downloaded file. The directory that opens as a result is referred to as the <PROMETHEUS_HOME> from here on.

To enable statistics for the Prometheus reporter, open the <SI_HOME>/conf/server/deployment.yaml file and set the enabled parameter in the wso2.metrics section to true, and update the other parameters in the section as shown below. You also need to add the metrics.prometheus: as shown.

 wso2.metrics:
   # Enable Metrics
   enabled: true
   reporting:
     console:
       - # The name for the Console Reporter
         name: Console

         # Enable Console Reporter
         enabled: false

         # Polling Period in seconds.
         # This is the period for polling metrics from the metric registry and printing in the console
         pollingPeriod: 2

 metrics.prometheus:
  reporting:
    prometheus:
      - name: prometheus
        enabled: true
        serverURL: "http://localhost:9005"

Open the <PROMETHEUS_HOME>/prometheus.yml file and add the following configuration in the scrape_configs: section.
```
 scrape_configs:
   - job_name: 'prometheus'
     static_configs:
     - targets: ['localhost:9005']
```
In the terminal, navigate to the <PROMETHEUS_HOME and issue the following command to start the Prometheus server.

./prometheus

Info

The above steps to configure and start Prometheus need to be performed before you start the Grafana server.

Downloading and setting up Grafana¶

The pre-configured dashboards provided by ESB Streaming Integrator which you previously downloaded are rendered in Grafana to visualize statistics. To download and set up Grafana, follow the steps below:

Before you begin:

Start the Prometheus server as instructed under Downloading and setting up Prometheus.

Download Grafana from the Grafana Labs - Download Grafana.
Start Grafana.

Info

The procedure to start Grafana depends on your operating system and the installation process. e.g., If your operating system is Mac OS and you have installed Grafana via Homebrew, you start Grafana by issuing the brew services start grafana command.
In the Data Sources section, click Add your first data source. In the Add data source page that appears, click Select for Prometheus.
In the Add data source page -> Settings tab, update the configurations for Prometheus as follows.
Click Default to make Prometheus the default data source.
Under HTTP, enter http://localhost:9090 as the URL.
Click Save & Test. If the data source is successfully configured, it is indicated via a message.
To import the dashboards that you previously downloaded as JSON files, follow the procedure below:
1. Start Grafana and access it via http://localhost:3000/.
2. To load a new dashboard, click the plus icon (+) in the side panel. Then click Import.
3. In the Import page, click Upload .json file. Then browse and select the .json file of the pre-configured dashboard that you downloaded (i.e., in step 5, substep 1).
4. If required, change the unique identifier displayed in the Unique Identifier (uid).
5. Click Import.

Enable the Siddhi application to publish statistics¶

To enable the SweetFactoryApp Siddhi application to publish statistics to Prometheus, add the @App:statistics(reporter = 'prometheus') annotation to it below the @App:name annotation as shown below:

Tip

You can update the Siddhi application in Streaming Integrator Tooling and deploy it again in the Streaming Integrator server as you did in Step 5: Update the Siddhi Application.

Viewing statistics¶

To generate some statistics and view them, follow the procedure below.

Start ESB Streaming Integrator.
To generate statistics, insert as many events as you want into the SweetProductionTable MySQL table that you created for this scenario in Step 1: Download Streaming Integrator and Dependencies. Also, manually add as many rows as you want in the /Users/foo/productioninserts.csv file.
Access Grafana via the localhost:3000 URL.
In the side panel, click the Dashboards icon and click Dashboards.

Then click on the ESB Streaming Integrator - Overall Statistics dashboard. It opens as follows.

Info

The statistics displayed will be different based on the number of records you inserted to the SweetProductionTable MySQL table and the number of rows you added in the /Users/foo/productioninserts.csv file during the last 30 minutes. You can also change the time interval for which statistics are displayed via the field for selecting the time interval in the top panel.
Under Overview Statistics, click SweetFactoryApp. The overview-statistics / ESB Streaming Integrator App Statistics dashboard opens.
Scroll down to the Sources section. The following is displayed.

The two entries displayed above represent the file source and the cdc source used in the SweetFactoryApp Siddhi application.
Scroll down further to the Destinations section. The file sink in the SweetFactoryApp Siddhi application is displayed as shown below.
Under Sources, click on the link to the productioninserts.csv file. The ESB Streaming Integrator - File Statistics dashboard opens. The contents of the productioninserts.csv file is the output of one query and the input of another. Therefore, it is a source as well as a destination, statistics are displayed for it under Source and Sink as shown below.

Source Statistics

Sink Statistics
Under ESB Streaming Integrator - File Statistics dashboard -> Sources, click on the file link. The file-statistics / ESB Streaming Integrator / File Source Statistics dashboard opens displaying detailed statistics for the file when it is functioning as a source.
Under ESB Streaming Integrator - File Statistics dashboard -> Sources, click on the file link. The file-statistics / ESB Streaming Integrator / File Sink Statistics dashboard opens displaying detailed statistics for the file when it is functioning as a source.
In the overview-statistics / ESB Streaming Integrator App Statistics dashboard -> CDC section, click on the SweetProductionTable link. The cdc-statistics / ESB Streaming Integrator / CDC Statistics dashboard opens with statistics generated for the cdc source in the SweetFactoryApp Siddhi application.

Under Streaming, click on the SweetProductionTable link. The cdc-statistics / ESB Streaming Integrator / CDC Streaming Statistics dashboard opens as follows.

What's Next?

To learn more about the key concepts of the Streaming Integrator component, see Streaming Integrator Key Concepts.
For more hands-on experience with Streaming Integrator component, try the Streaming Integrator Tutorials.
To learn about use cases specific to the Streaming Integrator component, see Streaming Integrator Use Cases.
Learn how to run ESB Streaming Integrator in containerized environments, try Running SI with Docker and Kubernetes

Top