Eventing using WSO2 DSS - Send data to a JMS Queue for a given event trigger

If you want to trigger an event depend on an SQL related service call this is a great example on how you can do it using WSO2 Data Service Server.

In this sample I am going to explain how you can handle event triggers in WSO2 Data service server.  In WSO2 DSS, you have two types of event triggers.

1. Input event triggers - which trigger's an event looking at the request.
2. Output event triggers - which trigger's an event looking at the response.

For both of these trigger's you can specify an ex-path expression to trigger the event.

To demonstrate this functionality I  am going to write a small data service which insert Student information. And every time an insertion happen, primary key of the inserted record will be send to a JMS Queue.

Prerequisites

In order to try out this sample you need to have
  -> WSO2 Data Services Server Downloaded
  -> You need to have a database of your choice
  -> Add related drivers to DSS_HOME/repository/component/lib folder
  -> ActiveMQ downloaded and started

Step 1 - Enable JMS senders in WSO2 DSS.

In order to send JMS messages from data services, you need to enable the JMS sender in data services server.
To enable JMSSender -> Go to DSS_HOME/repository/conf/axis2/axis2.xml and un-comment the following line.

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>

and then start the data services instance.

Step 2 - Create data Service.

If you are new on creating data services, please refer the following post on how to create a data service using WSO2 Data Service Server.

I have a small table called Student. Lets create a simple data service for the table student.

describe Student;
+---------+-------------+------+-----+---------+----------------+
| Field   | Type        | Null | Key | Default | Extra          |
+---------+-------------+------+-----+---------+----------------+
| sid     | int(11)     | NO   | PRI | NULL    | auto_increment |
| name    | varchar(90) | YES  |     | NULL    |                |
| address | varchar(75) | YES  |     | NULL    |                |
| country | varchar(75) | YES  |     | NULL    |                |
| phone   | varchar(20) | YES  |     | NULL    |                |
| major   | varchar(50) | YES  |     | NULL    |                |
| gpa     | float       | YES  |     | NULL    |                |
| tutorid | int(11)     | YES  |     | NULL    |                |
+---------+-------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

Log in to DSS management console -> Create New Data Service. Give the data service name appropriately and click on next.

Click on Add Data Source and give database configuration appropriately and click on next.

Next section we will discuss how we can write a data service Query which can invoke a trigger according to the query result.

In this Query my SQL is simple insert

INSERT INTO Student(name,address,country,phone,major,gpa,tutorid) VALUES(:name,:address,:country,:phone,:major,:gpa,:tutorid)

Click on "Generate Input Mappings", and this will generate the input mappings for the input parameters.

Since we need a response back (ideally it will be the primary key of the newly inserted record) we can click  Return generated key which will auto generate the response.

To add the event trigger, scroll down a little bit. Under Events, click on Manage Events. There, we give the eventing configurations.

Event ID - student_addition_trigger

Xpath - //*[local-name()='ID' and namespace-uri()='http://ws.wso2.org/dataservice']>0

Target Topic - student_insertion_topic

Event Sink URL ( JMS URL according to your activeMQ)

jms:/student_insertion_topic?transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=Content-Type&java.naming.provider.url=tcp://localhost:61616&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&transport.jms.ConnectionFactoryType=queue&transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory

Go to Main Configuration and add the Output trigger as shown below.

Now we have done with our Query Section. Click on Save to save the Query and Click next to add the  operation.
Add New Opperation ->
Operation Name* - InsertStudent
Query ID* - insertQ

Save and finish.

Now we can test this functionality by using the try it function provided by Data Service Server

My Request

   <body> <p:InsertStudent xmlns:p="http://ws.wso2.org/dataservice">
      <p:name>Amani</p:name>
      <p:address>Soysa</p:address>
      <p:country>SL</p:country>
      <p:phone>1923123111</p:phone>
      <p:major>SE</p:major>
      <p:gpa>4.2</p:gpa>
      <p:tutorid>12</p:tutorid>
   </p:InsertStudent>
</body>

My Response

   <GeneratedKeys xmlns="http://ws.wso2.org/dataservice"> <Entry>
      <ID>5</ID>
   </Entry>
</GeneratedKeys>

If you go to your active MQ console, You would see number of pending messages as shown below. You can go inside your Queue and explore more!!!!

Distributed Transaction with WSO2 Data Services

What is distributed transaction ?

Distributed database transaction means executing multiple related actions/operations in a coordinated way. This is also known as  global transaction. Distributed transaction can occur in the same database level however, in most cases distributed transaction happens in different databases (typically different RDBMS types) and often in different locations. Distributed transactions are often described as ACID -- atomic, consistent, isolated, and durable.  Meaning changes made to the database during the transactions are tentative, if any of the operation fails then none of the other changes will get affected. In a typical distributed transaction, you have to make sure if one operation fails the whatever previously executed operation should roll back, undoing all the changes as if the transactions never took place. Even if your application crashes in the middle of the transaction, when it restarts, transaction recovery should roll back the open transaction.

A typical Distributed transaction example would be moving data from one data base to another database. For example lets say you want to delete customer from one location and add that same customer to the another location.You would not want either transaction committed without assurance that both will complete successfully. Therefore, for these kind of instances its important to have distributed transaction feature.

The above transactions involve the following steps:

1. Begin a transaction. 
2. Add Customer
3. Delete Customer
4. End transaction

Distributed Transaction with WSO2 Data Services Server

WSO2 data services server provide distributed transaction using Java Transaction API (JTA) which enables global level transaction across multiple X/Open XA resources in java environment.

When you use XA functionality, the transaction manager uses XA resource instances to prepare and coordinate each transaction branch and then to commit or roll back all each of individual transaction appropriately.

For each RDBMS type there is a specific XA-Datasource class and set of configuration properties. Therefore you need to know the XA-Datasource class and their configurations before creating the data service.

Lets see how we can create a data service for the above transaction.

For this example I will have two databases in postgres and mysql. For this demo I will be using a very simple customer table which has id and name and we will see how we can inset values to these two tables in each database in a coordinated manner

=========================== SQL script ==============================

CREATE TABLE customer (
cust_id int NOT NULL,
name varchar(255) NOT NULL,
PRIMARY KEY (cust_id)
)

========================data service configuration file======================

<data disableStreaming="true" enableBoxcarring="true" enableDTP="true" name="DTPDS">
   <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
   <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
</data>

You need to add the configuration file to WSO2DS_HOME/repository/deployment/server/dataservices in order to deploy this file

Step by step explanation

Data source configuration

I have created two data source configuration in the data service descriptor file

 <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>

There we have specified the XA datasource classes "org.wso2.ws.dataservice.xa_datasource_class" in a property along with it's parameters. For each XA datasource class configuration properties may differ.

Query Configuration

  <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>

I have created two queries pointing each data source, and also two distinct opperations mapping to each query.

<operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>

After you deploy this data service. You will see the deployed data service under the data service list. You can invoke this service using the try-it tool (or using your own class)

Invoke the operations in this order

1. begin_boxcar 
2. my_insert
3. pos_insert
4. end_boxcar

Please make sure you have set the  "max_prepared_transactions" to a non zero value in "/etc/postgres/postgres.conf in oorder this sample to work

In my next post, I will be explaining how we can call all these four operations using a single proxy service with the use of of WSO2 ESB.

How to use WSO2 Payload mediator - Calling data service insertion using payload mediator

In my previous blog I showed how to use an iterate mediator to iterate through a soap message. In this post I am going to explain how WSO2 ESB payload mediator works.

Lets say you have a service which provides set of data and you want to call a data service insert operation. This message is generated from a data service, which access a database table and get set of records from the database. Please refer "How to create a MYSQL data service using WSO2 data services Server" If you want to create the data service and generate the below Request,

<Keys xmlns="http://ws.wso2.org/dataservice">
<Key>
    <P_Id>1</P_Id>
    <LastName>Soysa</LastName>
    <FirstName>Amani</FirstName>
    <Address>361 Kotte Road Nugegoda</Address>
    <City>Colombo</City>
 </Key>
 <Key>
    <P_Id>2</P_Id>
    <LastName>Bishop</LastName>
    <FirstName>Peter</FirstName>
    <Address>300 Technology BuildingHouston</Address>
    <City>London</City>
 </Key>
 <Key>
    <P_Id>3</P_Id>
    <LastName>Clark</LastName>
    <FirstName>James</FirstName>
    <Address>Southampton</Address>
    <City>London</City>
 </Key>
 <Key>
    <P_Id>4</P_Id>
    <LastName>Carol</LastName>
    <FirstName>Dilan</FirstName>
    <Address>A221 LSRC Box 90328 </Address>
    <City>Durham</City>
 </Key>
</Keys>

Lets see how we can use this data set which will come to WSO2 ESB as a soap request and we need to extract soap payload data and send them to a data service. First we need to use the iterate mediator  which will iterate the soap request if you have more than one data set. And we need to create the data service soap request using the payload mediator.

   <payloadFactory>
  <format>
     <p:InsertPerson xmlns:p="http://ws.wso2.org/dataservice">
        <p:P_Id>?</p:P_Id>
        <p:LastName>?</p:LastName>
        <p:FirstName>?</p:FirstName>
        <p:Address>?</p:Address>
        <p:City>?</p:City>
     </p:InsertPerson>
  </format>
  <args>
     <arg expression="//P_Id/text()" />
     <arg expression="//LastName/text()" />
     <arg expression="//FirstName/text()" />
     <arg expression="//Address/text()" />
     <arg expression="//City/text()" />
  </args>
</payloadFactory>

Once we create the payload mediator then we can create a send mediator to insert data to data service

  <send>

     <endpoint>

        <address uri="http://localhost:9765/services/MyFirstDSS/" />

     </endpoint>

  </send>

When you add everything together your proxy service will look like shown below.

<proxy xmlns="http://ws.apache.org/ns/synapse" name="AssetProxyService" transports="https,http" statistics="disable" trace="disable" startOnLoad="true">
  <target>
     <inSequence>
         <iterate xmlns:m="http://ws.wso2.org/dataservice" id="iter1" expression="//m:Keys/m:Key">
           <target>
              <sequence>
                 <payloadFactory>
                    <format>
                       <p:InsertPerson xmlns:p="http://ws.wso2.org/dataservice">
                          <p:P_Id>?</p:P_Id>
                          <p:LastName>?</p:LastName>
                          <p:FirstName>?</p:FirstName>
                          <p:Address>?</p:Address>
                          <p:City>?</p:City>
                       </p:InsertPerson>
                    </format>
                    <args>
                       <arg expression="//P_Id/text()" />
                       <arg expression="//LastName/text()" />
                       <arg expression="//FirstName/text()" />
                       <arg expression="//Address/text()" />
                       <arg expression="//City/text()" />
                    </args>
                 </payloadFactory>
                 <send>
                    <endpoint>
                       <address uri="http://localhost:9765/services/MyFirstDSS/" />
                    </endpoint>
                 </send>
              </sequence>
           </target>
        </iterate>
     </inSequence>
  </target>
  <description />
</proxy>

Expose your entire database as data services in two steps !!!

WSO2-Data Services Server provides the feature to expose your data as a service right away!!!. All you need to give is your database information and then generate data services to expose your database tables/schemas as per your need.

if you are using wso2 products for the first time ...

You need to download wso2 dataservices server.

Start up the server -> go to DS_HOME/bin/wso2server.bat | wso2server.sh (DS_HOME is where your set up is located)

Once the server is up and running open a web browser and navigate to https://localhost:9443/carbon.

Login it the server using the default credentials (username=admin, password=admin).

Also note that you need to provide the related jdbc driver for the database and put in repository/component/lib in order for this feature to work.

Step 1 :- Create a data source

In order to expose your database you need to create a data source using wso2 data services server.

To create data source click on Configure -> datasources

Click on add data source and give your data base information.

Once you create your data source you can test your connection by giving a validation query (ie Select 1) to make sure your connection is successful.

Step 2 Generating data services

You can start by clicking on the generate link in the main menue. Once you click on generate you will be directed to a wizard which will ask for the datasource and the database name which will be needed to expose.

Once you create you give your database name you will get the list of schemas/tables to select for your data service.

Click on next to view the created data services

Click on finish to view the deployed services.

You can try your service by clicking on try this sample for database transactions :) You can also generate axis2 clients by clicking on the data service and going to the service dashboard.

Once you create your data service there are features to enable security, throttling and caching by going to the service dashboard as shown below.

Try your service - using the try it tool.

Extracting RDF Data using WSO2 Data Services. (How to extract aircraft information from NASA rdf data sources)

In my last post I explained how to expose your data in cloud as RDF resources. Today I am going to explain how we can query RDF resources on the web/cloud and expose extracted data as a service. First of all as usual to expose our data as a service we need to create a data service using wso2 data services server.

To demonstrate RDF data extraction I am going to use a popular RDF data source which stores interesting information about NASA aircraft details. And we are going to extract aircraft information according to the agency. Following diagram shows how NASA keep their aircraft information as a data source.

If you click on RDF/XML on the top right hand side corner link you can view the RDF source of the the data.

To create data services you need to either download and install wso2 data services server or you can straight away create data services using Stratoslive cloud platform.

Once you login to data services server, click on create under Web Services -> Data Services -> add. Give an appropriate name and click on next.

Once you login to data services server, click on create under Web Services -> Data Services -> add. Give an appropriate name and click on next.

Then you need to create an RDF data source using our NASA rdf datasource. Give the DataSource Id, Data Source Type, and RDF File Location.

RDF File Location - http://nasa.dataincubator.org/~search.rdf?query=all

Click on next and create new query to create a new query. In order to query RDF data we need to write queries using SPARQL query language which is similar to SQL.The following SPARQL query is used to extract aircraft information

PREFIX space: <http://purl.org/net/schemas/space/> 
PREFIX relevance: <http://a9.com/-/opensearch/extensions/relevance/1.0/> 
PREFIX foaf: <http://xmlns.com/foaf/0.1/>       
PREFIX dc: <http://purl.org/dc/elements/1.1/>                                                 
SELECT ?homepage ?name ?alternateName ?internationalDesignator ?mass ?score ?launch ?agency ?description 
WHERE { 
  ?craft foaf:homepage ?homepage. 
  ?craft foaf:name ?name. 
  ?craft space:alternateName ?alternateName. 
  ?craft space:internationalDesignator ?internationalDesignator. 
  ?craft space:mass ?mass. 
  ?craft relevance:score ?score. 
  ?craft space:launch ?launch. 
  ?craft space:agency ?agency. 
  ?craft dc:description ?description. 
}

Click on add Input mapping to give input mapping parameters. Since we are going to give agency as a input parameter we need to specify it in our rdfQuery. And go to main configuration.

Give the Query information and the SPARQL query as shown below.

Now we need to arrange how we display the results in the Results (output mapping section).

• Output type – xml
• Grouped by element - Aircrafts
• Row name – Aircraft

Click on add new output mappings to add output mapping elements. Give the mapping type, output field name and data source column name as shown in the table below.

Mapping Type	Data Source Type	Data Source Column Name
element	homepage	homepage
element	name	name
element	alternateName	alternateName
element	internationalDesignator	internationalDesignator
element	mass	mass
element	score	score
element	launch	launch
element	agency	agency
element	description	description

Click on save to save the query. Now we'l add an operation to get our extracted data called getAircrafts. Click next -> add New Operation. And give the query name and the operation name. Click on finish to finish creating the data service.

Click on finish to deploy the data service. Once you click on finish you can see your deployed data service under service list as shown below.

To try this service click on our try-it feature. Lets test our service by giving “United States” as the input of our service. You can see we can get all the aircraft details coming from United States agency.

Expose your cloud data as RDF Resources

Since its all about semantic web 3.0 and RDF Data linking, I am going to explain about RDF data and exposing RDF data in the cloud space in 5 to 10 mins :) Just by using WSO2 Stratos Data Services Server.

The Resource Description Framework (RDF) is one of the most powerful technique to expose and interlink data(knowledge) in the decentralized world. It is also the latest trend in publishing and consuming linked data on the cloud therefore, lets discuss how we can expose our data as a RDF resource in the cloud using WSO2 Stratos Data Services Server.

1) use the RDF data model to publish structured data on the Web

RDF data model consist of set of statements which has a way of publishing link data on the web as triplets (with the use of subject predicate and object). In simple terms RDF model is a way of representing machine understandable data on the web as shown in the diagram below.

2. use RDF links to interlink data from different data sources

All things described by RDF are called resources, RDF links represents the linkage between one resource to another which is mainly done by the use of URIs.

-------------------Simple RDF file --------------

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns# xmlns:cd="http://www.product.fake/cd#">

<rdf:Description rdf:about="http://www.product.fake/cd/S10_1678 ">

<cd:productCode>S10_1678</cd:productCode>

<cd:productName>1969 Harley Davidson Ultimate Chopper</cd:productName>

<cd:productLine>Motorcycles</cd:productLine>

<cd:quantityInStock>7933</cd:quantityInStock>

<cd:buyPrice>48.81</cd:buyPrice>

</rdf:Description>

</rdf:RDF>

Now that we have a brief understanding on RDF and the importance of RDF data, lets see how we can generate RDF data source from a Google spread sheet.

First you need to create a google spread sheet of your choice which has some sensible information. To get the full usage of RDF you need to create several rdf

resource for the linking purposes however, for clarity purposes I will demonstrate how to create a single RDF resource and link it with an existing RDF resources.

Lets expose a google spreadsheet with product information on vehicle sales.

Product – Describe the currently available products in a car sale vendor.

ID	Model	Classification	Qty
S10_1678	1996 Moto Guzzi 1100i	Motorcycles	12
S10_1949	2003 Harley-Davidson Eagle Drag Bike	Classic Cars	23
S10_2016	1972 Alfa Romeo GTA	Motorcycles	18
S10_4698	1962 LanciaA Delta 16V	Motorcycles	15
S10_4757	1968 Ford Mustang	Classic Cars	13
S10_4962	2001 Ferrari Enzo	Classic Cars	12
S12_1099	1968 Ford Mustang	Classic Cars	4
S12_1108	2001 Ferrari Enzo	Classic Cars	10

Lets assume we have another set of RDF resources on product line ( which has information on each product line type) ie http://productLines/car , http://productLines/cycle, http://productLines/bus

Now lets create a data service to expose our Spreadsheet data as a rdf resource. In order to expose these data in the cloud you need to have a stratoslive account. Once

you create your stratos live account you can access set of stratos services such as Enterprise Service Bus, Application Server, Data Services etc (to try out stratos services you can easily create a demo account for free )

After creating your stratos account you can easily logged into your tenant domain and start working in the cloud!!!!

Now lets go back to exposing spreadsheet data as a service ... In order to do that we need to use WSO2 Stratos Data Services Server which provides a

powerful set of feature to expose data as a service and set of service utility methods. To access data services go to stratos live manager home page and click on wso2 stratos Data services.

To create a data service go to the left side menu bar and click on create under webservices->Add->Data Services. Then you will get a wizard as shown below. Give a proper data service name and click on next.

Once you click on next you will be directed to add data source page. And give information regarding the google spreadsheet you created along with your credentials

You can click on test connection to confirm your connection.

Click on next to go to the Query page. Query page describe the extracting algorithm to extract your data from the data source (google spreadsheet). Lets extract ProductID, Model,Classification and Qty.

Since our output is RDF result set, we need to specify our output type as RDF. RDF Base URI is the format of rdf:about URI which uniquely identifies each resource.

We will give RDF base URI as http://www.product/cd/{1}; this takes the Spreadsheet column 1(which is the ID) value for each row and replaces it for the RDF about attribute inside rdf:Description element

Output Type – RDF

RDF Base URI :- http://www.product/cd/{1}

Row namespace :- http://www.product/cd#

To generate the response in RDF format click on "Add New Output Mappings" button. There are two mapping types in RDF Output mapping. 1) as a element, 2) as a resource.When mapping an element as a resource, you need to give the resource URI along with the column name which needs to be mapped in curly brackets as shown below. This way we can link two RDF resources together and create a relationship between each other.

Lets map ID, Model and Qty as elements and Classification as a resouce, Lets link classification column to the productline resouces as i mention earlier ( http://productLines/car , http://productLines/cycle, http://productLines/bus )

Mappings of RDF resource

Resource URI http://productLines/{3} (as you can see we put the column 3 to get each classification type of the product).

Resouce Field Name - Classification

Mappings of RDF element

Following diagram shows the output mappings which we mapped from google spreadsheet to RDF resource.

Once we create the the query click on next to add Resources. Since we are exposing data as RDF resource we need to create a resource to expose the data. Lets give our query information when creating the resource.

Resouce Path – Products

Resource Method – Get

Query ID – RDFQuery

Click on finish to deploy the data service. Once you click on finish you can see your deployed data service under service list as shown below.

Now that we created our RDF resource we can test it by accessing it as a rest call or by using the try its feature.

Rest URL https://data.stratoslive.wso2.com/services/t/amani123.com/RDFDataSource/_getproducts (replace the tenant name amani123.com with your tenant domain)

You can validate this RDF resource by using the online RDF validator by copy pasting the rdf resource (right click on the page and view page source copy paste it inside the validation)

Now we exposed our spreadsheet data in the cloud space just within 10 mins :) you can create more rdf data sources using the same manner with different data sources (csv/excel/rdbms) and expose those data as RDF data sources. I will further explain how we can extract RDF data using SPARQL in my next blog post :)

Extracting Web information using WSO2 Data Services

This blog explain how to scrape web data using wso2 data services web harvesting feature. In this tutorial I am going to extract Top rated books from Top Rated Books - Book Movement web page along with their authors and expose those data as a data service.

Before I begin lets look at how scraping works.

When you scrape a web page you need to identify the html/xml pattern. If we look at Top Rated Books - Book Movement page you can see list of books are listed along with their details. Now if we look at the page source we can see several html tags are repeated in the same manner.

If you look at it closely you can see a wrapper element which is

<div class=”rgLayoutCenter”>

and inside that wrapper element you have some thing like this.

<div class="rgLayoutTitle">First They Killed My Father: A Daughter of Cambodia Remembers (P.S.)</div>  
<div class="rgLayoutAuthor">by Loung Ung</div>

Now our basic requirement is to extract all the books along with their authors. To do that we need to find the pattern of the book title as wel as the author.

<div class="rgLayoutTitle">First They Killed My Father: A Daughter of Cambodia Remembers (P.S.)</div>  
 <div class="rgLayoutAuthor">by Loung Ung</div>

we can easily write an xslt template to extract these information

<?xml version="1.0" encoding="ISO-8859-1"?> 
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> 
<xsl:output method="xml" omit-xml-declaration="yes" indent="yes"/> 
<xsl:template match="/"> 
  <BookInfo> 
   <xsl:for-each select="//div[@class='rgLayoutCenter']"> 
     <Book> 
       <Title><xsl:value-of select="div[@class='rgLayoutTitle']"/></Title> 
   <Author><xsl:value-of select="div[@class='rgLayoutAuthor']"/></Author> 
     </Book> 
   </xsl:for-each> 
  </BookInfo> 
</xsl:template> 
</xsl:stylesheet> 

Above xslt template describe to go inside each

<div class=”rgLayoutCenter”> 

element and extract value of

<div class="rgLayoutTitle">

and

<div class="rgLayoutAuthor">

and assign it to the XML elements Title and Author inside the wrapper Book element.

Creating the web harvest data service

Now we learnt the basic concepts on web scraping/web harvesting, we will straight away create the data services using WSO2 Data services server.

To install WSO2 data services download and unzip the zip file and go to $DS_HOME/bin and start up the server from the command prompt, run bin/wso2server.bat{sh}

When the server startup is complete, access http://localhost:9443/carbon in your browser. Sign in to the server using the default credentials (username=admin, password=admin) in the right hand side corner. It will redirect you to the management console page.

To create the data service click on create in the left hand side menu under Web Services -> Add -> Create

Once you click on it you will have to fill the data service information as shown below. Lets name our data service as WebHarvestDS.

Once you click on next you will be redirected to create data-source page click on add new datasource to add our web datasource.

For the web datasource you need to have a configuration file along with the web scraping url, the scraperVariable and the HTTP method to extract data.

Also we need to provide our template.xslt file location we wrote earlier.

<?xml version="1.0" encoding="UTF-8"?>
<config>
 <var-def name='bookInfo'>
  <xslt>
   <xml>
    <html-to-xml> 
     <http method='get' url='http://www.bookmovement.com/app/readingguide/memberRecommendations.php'/>
    </html-to-xml>
   </xml>
   <stylesheet>
    <file path="/media/ntfs/web/template.xsl"/>
   </stylesheet>
  </xslt>
 </var-def>
</config>

Place the above configuration file inside the inline configuration section (or you can save the above configuration in your local machine and give it as a web harvest config file path)

Lets give our scraper variable as bookInfo and HTTP method as get and also our template location.

Creating the Query

Click on next to add a query. Give a queryId, scraper variable as webQuery and bookInfo.

To populate the data properly we need to give the group by element and row-name. This is to tell the web service how our result format should be. We also need to give output mapping to arrange our extracted data.

Query information.

• QueryId – webQuery
• Data Source – web
• Scraper Variable-bookInfo
• Grouped by element – Books
• Row name – Book

output mappings

• Element – Title
• Element – Author

Click on Next to add operation and select the query name and give a name to our operation.

Click on finish to finish creating the data service. Go to webservices list and you can see our webservice is successfully deployed.

You can invoke the service using the try-it tool.

You can also invoke the data service using a rest call by simply typing http://localhost:9763/services/WebScraping/getBooks/ on ur browser.