Thing which seemed very Thingish inside you is quite different when it gets out into the open and has other people looking at it

Saturday, October 12, 2013

How to Receive Emails to WSO2 ESB

In one of my previous blogs, I have explained how we can send emails using WSO2 ESB, in this post I am going to explain how we receive emails to WSO2 ESB, so that you can collect data from in coming emails and do some processing and send them to data storage or any other end point.


Before we start ESB server, you need to enable mail transport listeners in ESB. In order to do that you need to go to WSO2_ESB_HOME/repository/conf/axis2/axis2.xml and un comment the following.
<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener" />

Then start the server.

Creating the proxy service

To create the proxy, go to add -> Proxy service -> Custom Proxy service. There we'l give the name as MyMailProzy.
And you need to give set of service parameters as listed below.

 <parameter name="transport.PollInterval">5</parameter>

   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>

And also under Transport Settings you need to enable mailto transport as shown in the below diagram.

Click on next and we will configure the in sequence.

There we will log the sender's email address by getting the sender email address to a property from the transports (get-property('transport', 'From')) and then using log mediator to log it

<inSequence xmlns="http://ws.apache.org/ns/synapse">

   <property name="senderAddress" expression="get-property('transport', 'From')" scope="default" type="STRING"/>
   <log level="full">
      <property name="Sender Address" expression="get-property('senderAddress')"/>
   </log>
   <drop/>
</inSequence>



Click on next to configure the out sequence.

In the out sequence we will be having a simple send mediator which will send the response back to the client.

The full ESB Configuration

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="MyMailProzy"
       transports="mailto"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="senderAddress" expression="get-property('transport', 'From')"/>
         <log level="full">
            <property name="Sender Address" expression="get-property('senderAddress')"/>
         </log>
         <drop/>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <parameter name="mail.pop3.host">pop.gmail.com</parameter>
   <parameter name="transport.PollInterval">5</parameter>
   <parameter name="mail.pop3.password">wso2mail</parameter>
   <parameter name="transport.mail.ContentType">text/plain</parameter>
   <parameter name="mail.pop3.socketFactory.port">995</parameter>
   <parameter name="mail.pop3.user">wso2esb.mail</parameter>
   <parameter name="mail.pop3.socketFactory.fallback">false</parameter>
   <parameter name="mail.pop3.port">995</parameter>
   <parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
   <parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
   <parameter name="transport.mail.Protocol">pop3</parameter>
   <description/>
</proxy>
                               

Sending the EMAIL


I have used my gmail to send the email to ESB, my to address will be wso2esb.mail@gmail.com, and you can customize your own subject, and the message body.

Once we send the email, the email will be received to ESB, and you will see the following logs

[2013-10-13 09:06:45,038]  INFO - LogMediator To: , From: mailto:wso2esb.mail@gmail.com, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: <CAMJOoU667WqnFbx-cCG=hgOy3E2i2qRqKaAUoAcETJE8a-2b+g@mail.gmail.com>, Direction: request, Sender Address = Amani Soysa <amani.soysa@gmail.com>, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">--001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/plain; charset=ISO-8859-1&#xd; &#xd; Sending emails to  WSO2 Enterprise Service Bus!!!!&#xd; &#xd; --001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/html; charset=ISO-8859-1&#xd; Content-Transfer-Encoding: quoted-printable&#xd; &#xd; &lt;div dir=3D"ltr">Sending emails to=A0&lt;span style=3D"color:rgb(68,68,68);fon=&#xd; t-family:arial,sans-serif;line-height:16px" >=A0&lt;/span>&lt;span style=3D"font-w=&#xd;eight:bold;color:rgb(68,68,68);font-family:arial,sans-serif;line-height:16p=&#xd;x" >WSO2 Enterprise Service Bus!!!!&lt;/span>=A0&lt;/div>&#xd;&#xd; --001a11c233ea18b53d04e8970f15--&#xd; </text></soapenv:Body></soapenv:Envelope>

How to call multiple soap operations using call out mediator - WSO2 ESB

In this post I will explain how we can call multiple operations inside the same proxy service using the  call out mediator in WSO2 ESB. For this sample I have used my distributed transaction sample where we need to call multiple operation in the same transaction.

In order to do this you need to have WSO2 ESB, as well as sample web service with multiple operations.  For that I will be using WSO2 DSS distributed transaction data service which I have explained in my previous blog post. Where we will insert customer information, in to multiple data bases in the same transaction.

Sample input 

<customer>
  <id>2</id>
  <name>Smith</name>
</customer>

Step 1 - Creating the proxy service.

To create the proxy, you need to start the ESB server under services -> add -> proxy service -> Custom Proxy service. Give an appropriate name. For this sample I am calling it MyTransactionProxy.


In the in sequence I will be calling several mediators.

  1. Log Mediator - To log the incoming request
  2. Property Mediators - To store the input parameters
  3. Payload Factory Mediator - to arrange the payload for each operation
  4. Call out mediator  - to call each operation of the given service 


Sample In sequence

<inSequence xmlns="http://ws.apache.org/ns/synapse">
   <log level="full">
      <property name="M1" value="***************HITTING Transaction PROXY****************"/>
   </log>
   <property name="OUT_ONLY" value="true"/>
   <property name="id" expression="//id/text()"/>
   <property name="name" expression="//name/text()"/>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">            
            <soapenv:Header/>            
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:begin_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:my_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:my_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">          
            <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>          
            <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>      
         </p:pos_insert>
      </format>
      <args>
         <arg expression="get-property('id')" evaluator="xml"/>
         <arg expression="get-property('name')" evaluator="xml"/>
      </args>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:pos_insert">
      <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
   <payloadFactory media-type="xml">
      <format>
         <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:dat="http://ws.wso2.org/dataservice">          
            <soapenv:Header/>          
            <soapenv:Body/>      
         </soapenv:Envelope>
      </format>
   </payloadFactory>
   <callout serviceURL="https://localhost:9445/services/DTPDS/" action="urn:end_boxcar">
      <source type="envelope"/>
      <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
   </callout>
</inSequence>


Here I have created payload factory mediator and a call out mediator to call each operation. In the call our service URL I have given the end point for my service. For example I have an operation called my_insert. So I created a payload factory mediator and defined the payload format inline as shown below (parameters for the operation is given as $1,$2 which are later replaced by the id, and the name properties.

<payloadFactory xmlns="http://ws.apache.org/ns/synapse" media-type="xml">
   <format>
      <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">                                                
         <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>                                                
         <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>                                  
      </p:my_insert>
   </format>
   <args>
      <arg expression="get-property('id')" evaluator="xml"/>
      <arg expression="get-property('name')" evaluator="xml"/>
   </args>
</payloadFactory>

Likewise, I have called each operation with the same format.

In the out sequence I have used an empty send mediator which will send back the response to the same service.

Complete proxy xml is attached below.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="TransactionProxy"
       transports="http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <log level="full">
            <property name="M1"
                      value="*************HITTING Transaction PROXY*************"/>
         </log>
         <property name="OUT_ONLY" value="true"/>
         <property name="id" expression="//id/text()"/>
         <property name="name" expression="//name/text()"/>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:begin_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:my_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:my_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:my_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">
                  <xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
                  <xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
               </p:pos_insert>
            </format>
            <args>
               <arg expression="get-property('id')" evaluator="xml"/>
               <arg expression="get-property('name')" evaluator="xml"/>
            </args>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:pos_insert">
            <source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <payloadFactory media-type="xml">
            <format>
               <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                 xmlns:dat="http://ws.wso2.org/dataservice">
                  <soapenv:Header/>
                  <soapenv:Body/>
               </soapenv:Envelope>
            </format>
         </payloadFactory>
         <callout serviceURL="https://localhost:9445/services/DTPDS/"
                  action="urn:end_boxcar">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
   <description/>
</proxy>
                               

Calling the proxy service

Here I am calling the proxy service using the curl tool.

curl -v --request POST -d '2Smith' -H Content-Type:"text/xml" http://amani-ThinkPad-T520:8280/services/MyTransactionProxy




Wednesday, October 9, 2013

Distributed Transaction with WSO2 Data Services

What is distributed transaction ?

Distributed database transaction means executing multiple related actions/operations in a coordinated way. This is also known as  global transaction. Distributed transaction can occur in the same database level however, in most cases distributed transaction happens in different databases (typically different RDBMS types) and often in different locations. Distributed transactions are often described as ACID -- atomic, consistent, isolated, and durable.  Meaning changes made to the database during the transactions are tentative, if any of the operation fails then none of the other changes will get affected. In a typical distributed transaction, you have to make sure if one operation fails the whatever previously executed operation should roll back, undoing all the changes as if the transactions never took place. Even if your application crashes in the middle of the transaction, when it restarts, transaction recovery should roll back the open transaction.



A typical Distributed transaction example would be moving data from one data base to another database. For example lets say you want to delete customer from one location and add that same customer to the another location.You would not want either transaction committed without assurance that both will complete successfully. Therefore, for these kind of instances its important to have distributed transaction feature.

The above transactions involve the following steps:
  1. Begin a transaction. 
  2. Add Customer
  3. Delete Customer
  4. End transaction

Distributed Transaction with WSO2 Data Services Server

WSO2 data services server provide distributed transaction using Java Transaction API (JTA) which enables global level transaction across multiple X/Open XA resources in java environment.

When you use XA functionality, the transaction manager uses XA resource instances to prepare and coordinate each transaction branch and then to commit or roll back all each of individual transaction appropriately.

For each RDBMS type there is a specific XA-Datasource class and set of configuration properties. Therefore you need to know the XA-Datasource class and their configurations before creating the data service.

Lets see how we can create a data service for the above transaction.

For this example I will have two databases in postgres and mysql. For this demo I will be using a very simple customer table which has id and name and we will see how we can inset values to these two tables in each database in a coordinated manner

=========================== SQL script ==============================
CREATE TABLE customer (
cust_id int NOT NULL,
name varchar(255) NOT NULL,
PRIMARY KEY (cust_id)
)

========================data service configuration file======================

<data disableStreaming="true" enableBoxcarring="true" enableDTP="true" name="DTPDS">
   <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
   <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
</data>

You need to add the configuration file to WSO2DS_HOME/repository/deployment/server/dataservices in order to deploy this file

Step by step explanation

Data source configuration

I have created two data source configuration in the data service descriptor file
 <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
There we have specified the XA datasource classes "org.wso2.ws.dataservice.xa_datasource_class" in a property along with it's parameters. For each XA datasource class configuration properties may differ.

Query Configuration

  <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>

I have created two queries pointing each data source, and also two distinct opperations mapping to each query.

<operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>

After you deploy this data service. You will see the deployed data service under the data service list. You can invoke this service using the try-it tool (or using your own class)

Invoke the operations in this order

  1. begin_boxcar 
  2. my_insert
  3. pos_insert
  4. end_boxcar

Please make sure you have set the  "max_prepared_transactions" to a non zero value in "/etc/postgres/postgres.conf in oorder this sample to work

In my next post, I will be explaining how we can call all these four operations using a single proxy service with the use of of WSO2 ESB.

Thursday, July 25, 2013

Configure HTTP Access Logging in WSO2 products


If you want to analyze your application's usage activities such as who is accessing your page, number of hits, errors ect, you can use the HTTP access log files monitor and analyze the above measures. WSO2 products uses embedded tomcat as it's primary runtime therefore, you can use apache tomcat access logger to monitor your activity and performance of the server as well as any errors that may be occurring. This post explain how you can configure HTTP access logging in WSO2 products in order to get efficient monitoring.


In WSO2 products you can customize the http access log configuration by editing the catalina-server.xml which is located {CARBON_HOME}/repository/conf/tomcat directory, which is the server descripter file for the embedded tomcat integration. In the catalina-server.xml, under valves you have the HTTP access log configuration as shown below
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_" suffix=".log"
               pattern="combined" />

The server access log records all requests processed by the server. You can modify the what to log and what not to log by customizing the pattern attribute.
In the pattern attribute we can define formatting layouts. A formatting layout identifying the various information fields from the request and response to be logged, or the word "common" or "combined" to select a standard format.
Values for the pattern attribute are made up of literal text strings, combined with pattern identifiers prefixed by the "%" character to cause replacement by the corresponding variable value from the current request and response. The following pattern codes are supported:

%a - Remote IP address
%A - Local IP address
%b - Bytes sent, excluding HTTP headers, or '-' if no bytes were sent
%B - Bytes sent, excluding HTTP headers
%h - Remote host name
%H - Request protocol
%l - Remote logical username from identd (always returns '-')
%m - Request method
%p - Local port
%q - Query string (prepended with a '?' if it exists, otherwise an empty string
%r - First line of the request
%s - HTTP status code of the response
%S - User session ID
%t - Date and time, in Common Log Format format
%u - Remote user that was authenticated
%U - Requested URL path
%v - Local server name
%D - Time taken to process the request, in millis
%T - Time taken to process the request, in seconds
%I - current request thread name (can compare later with stacktraces) 


In addition, the caller can specify one of the following aliases for commonly utilized patterns:

common - %h %l %u %t "%r" %s %b
combined - %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"

Please note that the optimized access does only support common and combined as the value for this attribute.

If you want to change the format you can modify the as shown below

 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_" suffix=".log"
               pattern="%h %l %u %t '%r' %s %b" />



You can further change the log file location by changing the directory, prefix is the name of the log file. By default logs are rotated daily and the date is appended to the log file name. However, you can disable the log rotation by adding the following attribute rotatable="false". If you want to not have the current date appended in the log file name however, need the date to the rotated log file you can add renameOnRotate="true" attribute the the valve configuration. Please refer Access Log Valve Attributes for the full list of supported attributes. 



Wednesday, May 8, 2013

How to enable tenant aware service logging using WSO2 ESB


Unlike in other WSO2 products (WSO2 Appserver, WSO2 Data Services,  ect) we will not be able to see the application/service level logs  in ESB by default in ESB. Mainly because in the current carbon logging framework is  tenantaware and synapse is not multi tenanted and it runs on its own worker thread pool and does not know aboout the carbon context. Since it uses an independent thread pool from the carbon context logging does not work as expected with NHTTP. Therefore, to overcome this we wrote a handler call TenantDomainSetter which will set the tenant infomation/application infomation in the carbon context of each request.

To enable this you need to go to CARBON_HOME/repository/conf/axis2/axis2.xml and add the TenantDomainSetter in the prefetch of the request path (PreDispatch).

<handler name="TenantDomainSetter" class="org.wso2.carbon.utils.logging.handler.TenantDomainSetter"/>

Once you do that restart WSO2 ESB then you will be able to see service specific logs in LogViewer (monitor -> View Application Logs) which will show the logs of ESB proxy service logs. However application log viewer DOES NOT filter out Sequances, APIs, which are components inside a proxy service ... it will ONLY list down the proxy services.

Application Logs


System Logs


Please note since we are setting the tenant domain for every request there will be an aditional overhead in ESB.. if you want to minimize the overhead its recomended to switch of TenantDomainSetter and use ESB.

Sunday, April 28, 2013

Kick start on Openstack with Devstack {Folsom}

Recently I was told to install openstack controller-compute node set ... and my first impression on it was piece of cake!! because you get couple of scripts and all you need to just run the script.. !! I WAS SOO WRONG!! yes it is just couple of script but you need to know how things work and there are soo many places where it can totally messed up your machine and your set up. Meaning you can easily get messed up!!! If you don't go in the right path it will take ages .. and you might even have to format your machine several times to get things clean and start all over again which is very time consuming.. So I ended up installing openstack for almost a week spending day and night on my precious development time!!!


So I decided to write a post on focusing on HOW TO INSTALL OPENSTACK? So other people want do the same mistake I did or take ages to get things done. If you know what you are doing then installing devstack only take one hour's time all you need to do is make your configuration right and run the scrip!!!


So before we begin I would like to give a small introduction to OpenStack to make things more clearer.

What is OpenStack ?

In simple terms OpenStack is an opensource cloud operating system ... It is like the fabric of the cloud. OpenStack was initially founded by rackspace and NASA in 2010 and its being growing ever since. The best way to get what OpenStack is by reading their mission which is very simple and direct

"To produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable."

OpenStack is written in purely written in python, true it use other systems like MYSQL and rabbitMQ but the core is written in python which makes things very efficient and fast. And also initially it had two main component 1) Nova - which is the compute side of it and the other one 2) swift - the data storage side of it.

Current OpenStack stable version {Folsom} have few core components


  • Openstack object storage (swift)
  • OpenStack image registry and delivery (Glance)
  • Openstack compute (nova)
  • Horizon (web UI)
  • Keystone (authentication)
  • Quantum/Melange (Network)


In future OpenStack is going to provide other compoents such Cellometer (Billing service), Cinder (block storage) and load balancer services. So that's enough about OpenStack lets see how we can install it. I will be installing OpenStack using devstack..

What is devstack ?


Well devstack is openstack development project (there are quite a few :) ) but I think devstack is the easiest :).  Devstack provides a documented shell script (which is written in bash) to build a complete OpenStack development environments. Originally created by Rackspace now maintained by OpenStack community. This supports both ubuntu and fedora (but today we will be focusing how we can install openstack in ubuntu 12.04).

Installing OpenStack  

In this installation I will be explaining how we can install OpenStack in Multi-Node environment.

Prerequisites 

Install ubuntu 12.04 and

Install git
     apt-get install -y git sudo

Network setting on each node

    You need to have a stack ip and also decide on what your Floating IP range (public IP range ) and Fixed IP range (private IP range) are.

This is my network configurations

  Static IP (controller ) - 10.100.0.30
  Floating (Public) IP range - 10.100.0.128/25
  Fixed (Private) IP range  - 192.168.16.0/25 

Go to /etc/network/interfaces and configure your network accordingly.

Controller node 

auto eth0
iface eth0 inet static
address 10.100.0.30
netmask 255.255.252.0
gateway 10.100.1.254

Compute Node

auto eth0
iface eth0 inet static
address 10.100.0.29
netmask 255.255.252.0
gateway 10.100.1.254



Before we begin we need to create  separate user to do things in a cleaner way.


groupadd stack
useradd -g stack -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

From here onward we will be doing all the operation from the stack user.

Once you create the user log out and login from the the stack user and download openstack using git. When you are downloading make sure you download the stable version (in my case its folsom) so you will not have any surprises. :)


sudo su stack # login from the user
cd # go to user home
git clone git://github.com/openstack-dev/devstack.git -b stable/folsom #download devstack stable version

Once you download devstack other things are pretty straight foward. Go to devstack folder and create localrc  file and add the configuration accordingly in each node.

Configuring and Installing OpenStack in Controller Node


vi localrc

HOST_IP=10.100.0.30
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.16.0/25 
FIXED_NETWORK_SIZE=126
FLOATING_RANGE=10.100.0.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
SERVICE_PASSWORD=password
SERVICE_TOKEN=poohdedoo

Once you create the file you can run the setup script using the following command.

./stack.sh

After a successful  installation you will get the following output


Horizon is now available at http://10.100.0.30/
Keystone is serving at http://10.100.0.30:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: openstack
This is your host ip: 10.100.0.30
stack.sh completed in 3443 seconds.


Configuring and Installing OpenStack in Compute Node


Just like the controller node you need to have the basic infrastructure in order to install the computer. Only change is there is the content of the localrc file in the compute node.


vi localrc

SERVICE_HOST=10.100.0.30
HOST_IP=10.100.0.29
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.16.0/25 
FIXED_NETWORK_SIZE=126
FLOATING_RANGE=10.100.0.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
SERVICE_PASSWORD=password
SERVICE_TOKEN=poohdedoo
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_HOST=$SERVICE_HOST
ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol,rabbit


That's it !!. Just like the controller node you need to run the setup script to execute the installation.

./stack.sh

Once you successfully install openstack in both nodes you can login to openstack using the horizon dashboard

 http://10.100.0.30/ and login according to the user credentials you have provided in your setup configuration.


You can run the exercise.sh run provided by devstack in order to test your set up which includes uploading/launching instances and creating user groups/keypairs and user rules. 

If you want to upload images manually using the script you can use the following commands

glance image-create --name mysql-cartridge-amd64  --is-public true --container-format ami --disk-format ami < /home/wso2/wso2s2-openstack-1.0.0/cartridges/mysql-cartridge-amd64.img

To see what you have uploaded 

source openrc demo demo
glance image-list

You can also see these images using the horizon dashboard and further do task like launching the instances, assigning public IPs create key pairs and assign rules to security groups ect. I am not going to details since its a very straight forward dashboard and you can easily do everything by following the dashboard.



But I would like to proudly attach my openstack dashboad :) for references sake :)
On a final note restarting openstack is pretty tricky however, you can use the devstack script to restart openstack.

Restarting OpenStack using Devstack

Run the following in order to own the shell


script /dev/null 

And execute the following script
./rejoin-stack.sh

inside devstack folder in order to restart openStack.

To get out of the screen use ctrl-a , ctrl-d

Thats it folks !!
Once you set it up its like having a data center in your basement you can test load balancers and you can do all those caching stuff and experiment new stuff its really a cool thing to have. All the things you do which was in a private cloud and you can point it to a public cloud very easily!!!

Friday, April 26, 2013

Carbon Logging framework - Conversion Characters for TenantAwarePatternLayout


Usually when users use a logging framework one of their primary requirement is to customize the log output format. In Log4j framework this is done using the layout components.
There are couple of predefined layouts comming from the log4j api however, if you want more customize layout with extra set of attributes you need to write your own custom layout extending the org.apache.log4j.PatternLayout class.

In WSO2 Products we do not use the default PatternLyout we have written our own layout to make sure our log format is service aware and multi-tenanted. Therefore we have our own set of conversion characters defined in our TenantAwarePatternLayout class.

Below table explains characters used in the TenantAwarePatternLayout and all other characters (which are comming from PatternLayout) which you can use in your custom pattern.


Conversion
Character
Description                                          
Coming from custom carbon layout
D used to output the name of the tenant domain which is currently logged in to the server (super tenant domain is carbon.super)
T used to output the tenant id - to uniquely identify each tenant tenant id can be
used in the layout (super tenant id is 0)
S used to output the server key - to uniquely identify each carbon server a server key
can be used (ie as,esb,dss )
U used to output the user name - current user who is logged into the server
A used to output the application name - service name or web application name of the
current running applicaiton
H used to output the host name
I used to output the carbon instance id- uuid which is generated for each carbon instance
to uniquely identify each instance
Coming from the log4j patternlayout
c used to output category(logger); a.b.c → %c{2} = a.b
(this can be also used as fully qualified name)
d used to output the date of the logging event. for example, %d{hh:mm:ss,sss}
or %d{dd mmm yyyy hh:mm:ss,sss}.
F
used to output the file name where the logging request was issued.
l used to output location information of the caller which generated the logging event.
L used to output the line number from where the logging request was issued.
m used to output the application supplied message associated with the logging event.
M used to output the method name where the logging request was issued.
n used to output the platform dependent line separator character or characters.
p used to output the priority of the logging event.
r used to output the number of milliseconds elapsed from the construction of
the layout until the creation of the logging event.
t used to output the name of the thread that generated the logging event.

Please note %C, %F, %l, %L, %M slow down program run!

To create your own ConversionPattern you can go to the log4j.properties file and edit ConversionPattern of the appender of your choice.

log4j.appender.CARBON_LOGFILE.layout.ConversionPattern=TID: [%T] [%S] [%d] %P%5p {%c} - %x %m {%c}%n