Thing which seemed very Thingish inside you is quite different when it gets out into the open and has other people looking at it

Tuesday, August 30, 2016

Technology For Education ..


For the next few years, I will be focusing more on technology for education and going to discuss how technology can play a major role in early childhood education. Also I will be talking about education psychology, technology for  children with special needs and a whole lot about Autism.
The main reason for this change is because, as educators/care givers/human beings, we need to be aware of  children with special needs and see the world from their point of view and always help them to achieve their best. For the next couple of years I will be researching more about educational platforms that can improve cognition, social and  communication, emotional intelligence and linguistic skills in special needs child. Also how they can learn how to learn and see how we can improve the traditional education system for the betterment of our children.




Wednesday, January 29, 2014

Real time log event analysis with WSO2 BAM(CEP features)

One of the major problems when it comes to managing/monitoring distributed systems is not being able to detect when your system is giving issues all of a sudden. Lets say you have like 100 servers and you need to detect when a fatal error appears in the system and you want to act right away. This is nearly impossible if you do not have proper monitoring tools.
WSO2 BAM with WSO2 CEP features has the perfect mechanism to monitor real time logs and alert it to relevant parties when an expectation/ suspicious error occurs.



In this sample I will be using WSO2 Appserver and WSO2 BAM in order to demonstrate the monitoring and alerting capabilities of WSO2 BAM 2.4.0.

  • WSO2 Appserver will be used to send log events to BAM.
  • WSO2 BAM will do all the monitoring and alerting when logs are captured.
Send Logs to BAM


To send logs all you need to is go to WSO2_AS_HOME/repository/conf/log4j.properties and add LOGEVENT to the root logger. And start Appserver. (please make sure your BAM server is up and running before starting the Appserver)

To test if the logs are successfully sent to BAM, you can log in to cassandra explorer and see if there is a new column family created under EVENT_KS keyspace

ie log.AS.0.2013.1.12

Configuring WSO2 BAM for REAL TIME analytics

In this demo, I will be sending an email if an ERROR log occurs in WSO2 Appserver.  Since we will be using mail transport in BAM to send email alerts to recipients, we have to enable mail transport in BAM. To do that go to repository->conf->axis2->axis2-client.xml and add email configurations.


<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">
       <parameter name="mail.smtp.from">wso2esb.mail@gmail.com</parameter>
       <parameter name="mail.smtp.user">wso2esb.mail</parameter>
       <parameter name="mail.smtp.password">wso2mail</parameter>
       <parameter name="mail.smtp.host">smtp.gmail.com</parameter>
       <parameter name="mail.smtp.port">587</parameter>
       <parameter name="mail.smtp.starttls.enable">true</parameter>
       <parameter name="mail.smtp.auth">true</parameter>
   </transportSender>

Here you can give your own email configurations. 

Restart/Start the BAM server.

Assuming the logs are getting published to BAM, lets see how we can capture these log events for real time analytics.

Step 1 - Creating Event Adapters

In order to do real time analytics we need to create an execution plan. For that we need two input adapters.


  1. Input adapter - to capture log events coming to BAM (in  this case it will be a wso2event).
  2. Output adapter - to send emails
To create adapters you need to login to BAM management console. Under configurations, there will be "Event Processor Configs" and add input event adapters and output event adapters as shown below.

Input Event adapter


Output Event Adapter


Step 1 - Creating Stream Definitions

Now that we have created event adapters, we need to create a stream definition to capture the LogEvent in order to do complex event processing. In the appserver log event, these are the attributes we have.

  • Meta Data
    • clientType {String}
  • Payload Data
    • tenantID  {String}
    • serverName {String}
    • appName  {String}
    • logTime  {Long}
    • priority {String}
    • message {String}
    • logger {String}
    • ip {String}
    • instance {String}
    • stacktrace {String}
To create the stream definition, go to main tab and under create stream definition you can create the log event stream as shown below.


Step 1 - Creating the Execution Plan

In the execution plan we will be specifying the input stream and writing a CEP Query (SQL Like Query)  for the event stream.

Go to Create Execution Plan, and give suitable name for the execution plan. Select the needed stream and give an alias. Click on import after selecting the stream. In our CEP Query we will analyze events and if an event has an error, we will send it to an output stream.

CEP QUERY

from LogEvents[priority == "ERROR"]
select message,stacktrace,serverName
insert into ExceptionStream


After creating the Query, we need to add the exported stream (which is the stream that we are sending Error logs) according to our CEP querry exported stream name should be ExceptionStream give the value of the exported stream name and select "Create Stream Definition" to create the ExceptionStream. This stream will be auto generated by looking at the CEP as shown below.


Once we create the Exception stream select the Exception stream as exported stream and create a new formatter. This formatter is used to specify the email body and email related information like the subject, to address ect.  Give the output mapping type as text so we can give the content of the email message body inline. 


Email Body

Error Occurred in {{serverName}} – {{message}}
{{stacktrace}} 

In this body we are taking message, stacktrace and server name from the OutputStream (ExceptionStream) and adding a readable message for the email message body.

Add the event formatter and save the execution plan. Now we have successfully created the event trigger to monitory error logs for wso2 appserver. You can test this by invoking a service with an error.

If you want more in depth information on real time log event analytics you can follow the following screen cast for more details

Wednesday, January 15, 2014

[Screencast] Monitoring system logs with WSO2 Business Activity Monitor

This screencast explain how you can do your enterprise monitoring using BAM and CEP features.



Overview of the screencast

  * Send Log Events to BAM/CEP

  * Create analytics to analyse logs daily( can be scheduled accordingly)

  * Create gadgets with  log statistics (charts/tables)

  * Monitor logs real time with CEP (send alerts to relevant parties when an error log appears.)



Thursday, January 9, 2014

Eventing using WSO2 DSS - Send data to a JMS Queue for a given event trigger

If you want to trigger an event depend on an SQL related service call this is a great example on how you can do it using WSO2 Data Service Server.

In this sample I am going to explain how you can handle event triggers in WSO2 Data service server.  In WSO2 DSS, you have two types of event triggers.

1. Input event triggers - which trigger's an event looking at the request.
2. Output event triggers - which trigger's an event looking at the response.

For both of these trigger's you can specify an ex-path expression to trigger the event.

To demonstrate this functionality I  am going to write a small data service which insert Student information. And every time an insertion happen, primary key of the inserted record will be send to a JMS Queue.


Prerequisites

In order to try out this sample you need to have
  -> WSO2 Data Services Server Downloaded
  -> You need to have a database of your choice
  -> Add related drivers to DSS_HOME/repository/component/lib folder
  -> ActiveMQ downloaded and started

Step 1 - Enable JMS senders in WSO2 DSS.

In order to send JMS messages from data services, you need to enable the JMS sender in data services server.
To enable JMSSender -> Go to DSS_HOME/repository/conf/axis2/axis2.xml and un-comment the following line.

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>


and then start the data services instance.

Step 2 - Create data Service.

If you are new on creating data services, please refer the following post on how to create a data service using WSO2 Data Service Server.

I have a small table called Student. Lets create a simple data service for the table student.

describe Student;
+---------+-------------+------+-----+---------+----------------+
| Field   | Type        | Null | Key | Default | Extra          |
+---------+-------------+------+-----+---------+----------------+
| sid     | int(11)     | NO   | PRI | NULL    | auto_increment |
| name    | varchar(90) | YES  |     | NULL    |                |
| address | varchar(75) | YES  |     | NULL    |                |
| country | varchar(75) | YES  |     | NULL    |                |
| phone   | varchar(20) | YES  |     | NULL    |                |
| major   | varchar(50) | YES  |     | NULL    |                |
| gpa     | float       | YES  |     | NULL    |                |
| tutorid | int(11)     | YES  |     | NULL    |                |
+---------+-------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

Log in to DSS management console -> Create New Data Service. Give the data service name appropriately and click on next.

Click on Add Data Source and give database configuration appropriately and click on next.


Next section we will discuss how we can write a data service Query which can invoke a trigger according to the query result.

In this Query my SQL is simple insert

INSERT INTO Student(name,address,country,phone,major,gpa,tutorid) VALUES(:name,:address,:country,:phone,:major,:gpa,:tutorid)

Click on "Generate Input Mappings", and this will generate the input mappings for the input parameters.

Since we need a response back (ideally it will be the primary key of the newly inserted record) we can click  Return generated key which will auto generate the response.




To add the event trigger, scroll down a little bit. Under Events, click on Manage Events. There, we give the eventing configurations.

Event ID - student_addition_trigger

Xpath - //*[local-name()='ID' and namespace-uri()='http://ws.wso2.org/dataservice']>0

Target Topic - student_insertion_topic

Event Sink URL ( JMS URL according to your activeMQ)
jms:/student_insertion_topic?transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=Content-Type&java.naming.provider.url=tcp://localhost:61616&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&transport.jms.ConnectionFactoryType=queue&transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory

Go to Main Configuration and add the Output trigger as shown below.


Now we have done with our Query Section. Click on Save to save the Query and Click next to add the  operation.
Add New Opperation ->
Operation Name* - InsertStudent
Query ID* - insertQ

Save and finish.

Now we can test this functionality by using the try it function provided by Data Service Server

My Request


   <body> <p:InsertStudent xmlns:p="http://ws.wso2.org/dataservice">
      <p:name>Amani</p:name>
      <p:address>Soysa</p:address>
      <p:country>SL</p:country>
      <p:phone>1923123111</p:phone>
      <p:major>SE</p:major>
      <p:gpa>4.2</p:gpa>
      <p:tutorid>12</p:tutorid>
   </p:InsertStudent>
</body>

My Response


   <GeneratedKeys xmlns="http://ws.wso2.org/dataservice"> <Entry>
      <ID>5</ID>
   </Entry>
</GeneratedKeys>

If you go to your active MQ console, You would see number of pending messages as shown below. You can go inside your Queue and explore more!!!!





Saturday, October 12, 2013

How to Receive Emails to WSO2 ESB

In one of my previous blogs, I have explained how we can send emails using WSO2 ESB, in this post I am going to explain how we receive emails to WSO2 ESB, so that you can collect data from in coming emails and do some processing and send them to data storage or any other end point.


Before we start ESB server, you need to enable mail transport listeners in ESB. In order to do that you need to go to WSO2_ESB_HOME/repository/conf/axis2/axis2.xml and un comment the following.
<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener" />

Then start the server.

Creating the proxy service

To create the proxy, go to add -> Proxy service -> Custom Proxy service. There we'l give the name as MyMailProzy.
And you need to give set of service parameters as listed below.

 <parameter name="transport.PollInterval">5</parameter>

   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>

And also under Transport Settings you need to enable mailto transport as shown in the below diagram.

Click on next and we will configure the in sequence.

There we will log the sender's email address by getting the sender email address to a property from the transports (get-property('transport', 'From')) and then using log mediator to log it

<inSequence xmlns="http://ws.apache.org/ns/synapse">

   <property name="senderAddress" expression="get-property('transport', 'From')" scope="default" type="STRING"/>
   <log level="full">
      <property name="Sender Address" expression="get-property('senderAddress')"/>
   </log>
   <drop/>
</inSequence>



Click on next to configure the out sequence.

In the out sequence we will be having a simple send mediator which will send the response back to the client.

The full ESB Configuration

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MyMailProzy"
       transports="mailto"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="senderAddress" expression="get-property('transport', 'From')"/>
         <log level="full">
            <property name="Sender Address" expression="get-property('senderAddress')"/>
         </log>
         <drop/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="transport.PollInterval">5</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <description/>
</proxy>
                               

Sending the EMAIL


I have used my gmail to send the email to ESB, my to address will be wso2esb.mail@gmail.com, and you can customize your own subject, and the message body.

Once we send the email, the email will be received to ESB, and you will see the following logs

[2013-10-13 09:06:45,038]  INFO - LogMediator To: , From: mailto:wso2esb.mail@gmail.com, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: <CAMJOoU667WqnFbx-cCG=hgOy3E2i2qRqKaAUoAcETJE8a-2b+g@mail.gmail.com>, Direction: request, Sender Address = Amani Soysa <amani.soysa@gmail.com>, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">--001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/plain; charset=ISO-8859-1&#xd; &#xd; Sending emails to  WSO2 Enterprise Service Bus!!!!&#xd; &#xd; --001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/html; charset=ISO-8859-1&#xd; Content-Transfer-Encoding: quoted-printable&#xd; &#xd; &lt;div dir=3D"ltr">Sending emails to=A0&lt;span style=3D"color:rgb(68,68,68);fon=&#xd; t-family:arial,sans-serif;line-height:16px" >=A0&lt;/span>&lt;span style=3D"font-w=&#xd;eight:bold;color:rgb(68,68,68);font-family:arial,sans-serif;line-height:16p=&#xd;x" >WSO2 Enterprise Service Bus!!!!&lt;/span>=A0&lt;/div>&#xd;&#xd; --001a11c233ea18b53d04e8970f15--&#xd; </text></soapenv:Body></soapenv:Envelope>

How to call multiple soap operations using call out mediator - WSO2 ESB

In this post I will explain how we can call multiple operations inside the same proxy service using the  call out mediator in WSO2 ESB. For this sample I have used my distributed transaction sample where we need to call multiple operation in the same transaction.

In order to do this you need to have WSO2 ESB, as well as sample web service with multiple operations.  For that I will be using WSO2 DSS distributed transaction data service which I have explained in my previous blog post. Where we will insert customer information, in to multiple data bases in the same transaction.

Sample input 

<customer>
  <id>2</id>
  <name>Smith</name>
</customer>

Step 1 - Creating the proxy service.

To create the proxy, you need to start the ESB server under services -> add -> proxy service -> Custom Proxy service. Give an appropriate name. For this sample I am calling it MyTransactionProxy.


In the in sequence I will be calling several mediators.

  1. Log Mediator - To log the incoming request
  2. Property Mediators - To store the input parameters
  3. Payload Factory Mediator - to arrange the payload for each operation
  4. Call out mediator  - to call each operation of the given service 


Sample In sequence

<inSequence xmlns="http://ws.apache.org/ns/synapse">
   <log level="full">
      <property name="M1" value="***************HITTING Transaction PROXY****************"/>
   </log>
   <property name="OUT_ONLY" value="true"/>
   <property name="id" expression="//id/text()"/>
   <property name="name" expression="//name/text()"/>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">            
            <soapenv:Header/>            
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:begin_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:my_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:my_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:pos_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:pos_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">          
            <soapenv:Header/>          
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:end_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
</inSequence>


Here I have created payload factory mediator and a call out mediator to call each operation. In the call our service URL I have given the end point for my service. For example I have an operation called my_insert. So I created a payload factory mediator and defined the payload format inline as shown below (parameters for the operation is given as $1,$2 which are later replaced by the id, and the name properties.

<payloadFactory xmlns="http://ws.apache.org/ns/synapse" media-type="xml">
   <format>
      <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">                                                
         <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>                                                
         <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>                                  
      </p:my_insert>
   </format>
   <args>
      <arg expression="get-property('id')" evaluator="xml"/>
      <arg expression="get-property('name')" evaluator="xml"/>
   </args>
</payloadFactory>

Likewise, I have called each operation with the same format.

In the out sequence I have used an empty send mediator which will send back the response to the same service.

Complete proxy xml is attached below.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="TransactionProxy"
       transports="http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full">
            <property name="M1"
                      value="*************HITTING Transaction PROXY*************"/>
         </log>
         <property name="OUT_ONLY" value="true"/>
         <property name="id" expression="//id/text()"/>
         <property name="name" expression="//name/text()"/>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:begin_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:my_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:my_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:pos_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:pos_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:end_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
                               

Calling the proxy service

Here I am calling the proxy service using the curl tool.

curl -v --request POST -d '2Smith' -H Content-Type:"text/xml" http://amani-ThinkPad-T520:8280/services/MyTransactionProxy




Wednesday, October 9, 2013

Distributed Transaction with WSO2 Data Services

What is distributed transaction ?

Distributed database transaction means executing multiple related actions/operations in a coordinated way. This is also known as  global transaction. Distributed transaction can occur in the same database level however, in most cases distributed transaction happens in different databases (typically different RDBMS types) and often in different locations. Distributed transactions are often described as ACID -- atomic, consistent, isolated, and durable.  Meaning changes made to the database during the transactions are tentative, if any of the operation fails then none of the other changes will get affected. In a typical distributed transaction, you have to make sure if one operation fails the whatever previously executed operation should roll back, undoing all the changes as if the transactions never took place. Even if your application crashes in the middle of the transaction, when it restarts, transaction recovery should roll back the open transaction.



A typical Distributed transaction example would be moving data from one data base to another database. For example lets say you want to delete customer from one location and add that same customer to the another location.You would not want either transaction committed without assurance that both will complete successfully. Therefore, for these kind of instances its important to have distributed transaction feature.

The above transactions involve the following steps:
  1. Begin a transaction. 
  2. Add Customer
  3. Delete Customer
  4. End transaction

Distributed Transaction with WSO2 Data Services Server

WSO2 data services server provide distributed transaction using Java Transaction API (JTA) which enables global level transaction across multiple X/Open XA resources in java environment.

When you use XA functionality, the transaction manager uses XA resource instances to prepare and coordinate each transaction branch and then to commit or roll back all each of individual transaction appropriately.

For each RDBMS type there is a specific XA-Datasource class and set of configuration properties. Therefore you need to know the XA-Datasource class and their configurations before creating the data service.

Lets see how we can create a data service for the above transaction.

For this example I will have two databases in postgres and mysql. For this demo I will be using a very simple customer table which has id and name and we will see how we can inset values to these two tables in each database in a coordinated manner

=========================== SQL script ==============================
CREATE TABLE customer (
cust_id int NOT NULL,
name varchar(255) NOT NULL,
PRIMARY KEY (cust_id)
)

========================data service configuration file======================

<data disableStreaming="true" enableBoxcarring="true" enableDTP="true" name="DTPDS">
   <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
   <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
</data>

You need to add the configuration file to WSO2DS_HOME/repository/deployment/server/dataservices in order to deploy this file

Step by step explanation

Data source configuration

I have created two data source configuration in the data service descriptor file
 <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
There we have specified the XA datasource classes "org.wso2.ws.dataservice.xa_datasource_class" in a property along with it's parameters. For each XA datasource class configuration properties may differ.

Query Configuration

  <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>

I have created two queries pointing each data source, and also two distinct opperations mapping to each query.

<operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>

After you deploy this data service. You will see the deployed data service under the data service list. You can invoke this service using the try-it tool (or using your own class)

Invoke the operations in this order

  1. begin_boxcar 
  2. my_insert
  3. pos_insert
  4. end_boxcar

Please make sure you have set the  "max_prepared_transactions" to a non zero value in "/etc/postgres/postgres.conf in oorder this sample to work

In my next post, I will be explaining how we can call all these four operations using a single proxy service with the use of of WSO2 ESB.